Source-linked AI summary
PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading
Duong Hien Chi Kien, Thanh Trung Huynh
TL;DR
Trading RL must balance upside participation with drawdown control while accounting for market and portfolio state. PPO-HRAP blends PPO actions with an interpretable regime-derived exposure prior. In the main SPY run it ranks highest among compared methods on total return and risk ratios, while cross-asset transfers are supported by single-run evidence.
Problem
Trading RL can favor passive long exposure under return-only rewards or become excessively defensive under aggressive risk penalties, complicating the balance between upside and drawdown control.
Source: §1 · p. 1Method
PPO-HRAP combines PPO with an interpretable trend- and stress-based target exposure, blending the target with the learned action.
Source: §1 · p. 1Results
Takeaways & Limitations
The findings support the practical value of hybrid control-plus-learning policies in financial RL.
Source: §6 · p. 5Turnover is high, and cross-asset evidence for QQQ and DIA remains single-run and should be interpreted cautiously.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Reinforcement learning for trading often struggles to balance upside participation with drawdown control. Profit-only policies can collapse toward passive long exposure on upward-drifting assets, while aggressively risk-penalized rewards can become too defensive during volatile periods. This paper proposes PPO-HRAP, a hybrid regime-aware policy that combines Proximal Policy Optimization with an interpretable regime prior. The agent observes both market features and portfolio-state variables, receives a reward combining portfolio log return, VIX-conditioned drawdown-increase penalty, target-exposure deviation, and turnover cost, and executes a blended action between the PPO actor output and a regime-derived target exposure. On the held-out 2020-2022 SPY test window, PPO-HRAP achieves 27.62% total return, 8.48% annualized return, 0.6447 Sharpe ratio, 0.8588 Sortino ratio, and 0.4592 Calmar ratio, while reducing maximum drawdown from 34.10% for Buy and Hold to 18.47%. Across five SPY seeds, PPO-HRAP remains stable with mean total return $0.2725 \pm 0.0109$ and mean Sharpe ratio $0.6219 \pm 0.0565$. Single-run cross-asset tests on QQQ and DIA further show that the proposed method ranks first on total return and Sharpe ratio for all three reported assets. These results suggest that blending learned actions with a volatility-aware regime prior is a practical way to improve risk-adjusted trading behavior, although the current policy still incurs high turnover and cross-asset robustness beyond SPY remains limited to single-run evidence.
1 Introduction
Trading policies must balance upside participation with drawdown control while accounting for both market conditions and portfolio state. PPO-HRAP combines a regime-derived exposure prior with a learned PPO policy; its main SPY run leads the compared methods on total return and risk ratios.
1 Introduction
- Return-only policies can approach passive long exposure on upward-drifting indices, while aggressive risk penalties can sacrifice much of the upside.The paper frames trading as closed-loop allocation: actions change portfolio state and future risk, so useful agents need both market and portfolio information.Source: §1 · p. 1
- VIX can signal how aggressively to size a portfolio but provides no strong directional prior, leaving mild penalties ineffective and strong penalties potentially near-flat or short-biased.The design separates directional exposure, guided by trend, from risk sizing, guided by the VIX z-score.Source: §1 · p. 1
- PPO-HRAP computes an interpretable target exposure from trend and stress indicators, then blends it with the PPO action rather than relying on a fully unconstrained policy.Its reward combines portfolio log return, dynamic drawdown control, target-exposure regularization, and turnover control.Source: §1 · p. 1
- On the 2020–2022 SPY test window, PPO-HRAP achieves the highest total return and Sharpe, Sortino, and Calmar ratios among compared methods in the main run.The evaluation also covers five-seed SPY reliability and single-run transfers to QQQ and DIA; across seeds, SAC with a profit-only reward has better risk ratios at lower total return.Source: §1 · p. 2
2 Related Work
Related work spans policy-based trading and portfolio allocation, PPO and SAC optimization baselines, and risk-aware reward design. PPO-HRAP is presented as a hybrid of learned control and a structured regime prior, addressing the limits of reward shaping alone in specifying directional guidance.
2 Related Work
- Policy-based trading work includes PPO and SAC as continuous-control comparators, while classical allocation approaches emphasize the longstanding return-risk-adaptation trade-off.PPO is described as a practical backbone for continuous position weights, using a clipped surrogate objective to constrain large policy updates.Source: §2 · p. 2
- Risk-aware reward shaping can penalize volatility, downside risk, or drawdown, but does not necessarily specify which market direction to favor when stress rises.PPO-HRAP retains drawdown-aware shaping and adds a regime-derived action prior.Source: §2 · p. 2
- Drawdown is path dependent, so the simulator includes current drawdown and running peak in its state and penalizes increases rather than past stress itself.This distinguishes policies with similar final returns but different interim losses and investor experience.Source: §2 · p. 2
- PPO-HRAP is framed as a hybrid control-plus-RL design: its executed action depends on both a deterministic regime prior and the PPO policy output.The reported gains should therefore be interpreted as the joint effect of the prior and learned correction.Source: §2 · p. 2
3 Methodology
PPO-HRAP formulates ETF allocation around both market conditions and portfolio state, then combines a regime-derived exposure target with PPO actions and a reward that balances returns, drawdown, regime consistency, and turnover.
3.1 Problem formulation
- PPO-HRAP models daily ETF allocation as a discounted Markov decision process, with continuous portfolio actions and a state that includes exposure, cash, unrealized PnL, and drawdown.Including portfolio variables lets the policy account for how the same price movement affects portfolios with different exposures.Source: §3.1 · pp. 2–3
3.2 Feature engineering
- The actor receives 13 inputs: nine market features and four portfolio features, including short- and medium-horizon returns, trend, momentum, dispersion, and VIX stress signals.The feature set uses information available before each trading action.Source: §3.2 · p. 3
3.3 Regime prior and action blending
- Trend direction and VIX stress define a target exposure that PPO-HRAP blends with the actor’s action, keeping learned allocation adjustments while biasing execution toward a regime-consistent weight.The blend coefficient ranges from raw PPO action at ρ = 0 to a deterministic regime rule at ρ = 1.Source: §3.3 · pp. 3–4
3.4 Reward design and optimization
- The reward combines portfolio log return, drawdown increase, deviation from target exposure, and turnover; its drawdown penalty scales with VIX stress.Penalizing drawdown increases targets new deterioration relative to the running peak, while the turnover term discourages excessive reallocations where transaction costs apply.Source: §3.4 · p. 4
- During inference, the environment executes trades and logs reward for evaluation, but reward is not used to update the policy.The same feature computation, regime detector, action blending, and execution steps are used at inference.Source: §3.1 · p. 3, §3.4 · p. 4
4 Results
PPO-HRAP leads the main SPY run on reported return and risk-adjusted measures while reducing drawdown versus Buy and Hold; multi-seed and cross-asset results qualify that lead.
4.1 Experimental protocol and baselines
- Evaluation uses chronological 2010–2017 training, 2018–2019 validation, and 2020–2022 testing, with a 0.1% transaction cost and five seeds for the SPY reliability study.The main SPY result uses seed 42; QQQ and DIA are evaluated for cross-asset transfer.Source: §4.1 · p. 4
4.2 Main SPY performance
- Sharpe improves from 0.3420 to 0.6447 and maximum drawdown decreases from 34.10% to 18.47% versus Buy and Hold; PPO-HRAP also has the strongest reported SPY returns and risk ratios.Figure 2 compares total return against maximum drawdown.Source: §4.2 · pp. 4–5, Figure 2 · p. 5
4.3 SPY multi-seed reliability
- Across five seeds, PPO-HRAP has the best mean total return among RL methods, while SAC profit has stronger mean Sharpe, Calmar, and drawdown at lower mean return.The authors characterize PPO-HRAP as a return-leading hybrid, not a universal winner on every risk metric.Source: §4.3 · p. 5
4.4 Cross-asset evaluation
- In single-run tests, PPO-HRAP ranks first in total return and Sharpe for SPY, QQQ, and DIA, but SAC profit has the best QQQ Calmar.PPO-HRAP also ranks first in Calmar for SPY and DIA.Source: §4.4 · p. 5
4.5 Action-prior sensitivity and behavior
- Validation selects ρ=0.6, but performance is non-monotonic; PPO-HRAP lowers average exposure from 0.8839 in low-VIX periods to 0.4337 in high-VIX periods and remains full-long 54.10% of the test period.A weak prior may not sufficiently regularize the actor, while an overly strong prior can suppress learned correction.Source: §4.5 · p. 5
5 Discussion
PPO-HRAP blends a regime-derived exposure target with PPO actions, reacting to high VIX while retaining substantial full-long exposure. The discussion flags high turnover and limited cross-asset evidence as important trade-offs.
5 Discussion
- Blending the learned action with a regime-derived target gives PPO an interpretable prior tied to market stress and trend direction while leaving room to fine-tune execution.The paper frames the design as a hybrid control-plus-RL policy rather than a purely learned reward innovation.Source: §5 · p. 5
- Average exposure falls from 0.8839 in low-VIX periods to 0.4337 in high-VIX periods, yet the policy remains fully long for 54.10% of the test period.The authors describe the regime prior as a partial correction to vanilla PPO behavior, not a complete solution to long bias.Source: §5 · p. 5
- SPY test turnover is 51.22 and cross-asset turnover remains around 39–56; QQQ and DIA evidence is single-run, and training uses 30,000 timesteps.The authors present the results as a benchmark snapshot rather than a fully saturated optimization study.Source: §5 · p. 5
6 Conclusion
The paper concludes that PPO-HRAP combines dynamic drawdown control, regime-guided target exposure, and action blending for risk-controlled trading. Its SPY, multi-seed, and single-run cross-asset results are presented as promising, with broader evaluation left as future work.
6 Conclusion
- In the 2020–2022 SPY run, PPO-HRAP improves total return and risk-adjusted metrics over financial and RL baselines while reducing maximum drawdown relative to Buy and Hold.The conclusion identifies Table 5 as the held-out SPY test comparison.Source: §6 · p. 5, Table 5 · p. 6
- Multi-seed SPY results show that the reported gains are not confined to a single random seed.Table 6 reports the SPY multi-seed test summary across RL methods.Source: §6 · p. 5, Table 6 · p. 6
- Single-run transfers to QQQ and DIA indicate that the hybrid regime prior can remain useful beyond the original market.Table 7 reports single-run QQQ and DIA test performance.Source: §6 · p. 5, Table 7 · p. 6
- Broader multi-seed cross-asset evaluation, longer training, deeper turnover analysis, richer regime definitions, and testing across more assets and market conditions remain next steps.The conclusion identifies these as priorities for future work.Source: §6 · p. 5