Source-linked AI summary
Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading
Arishi Orra, Himanshu Choudhary, Manoj Thakur
TL;DR
Stock-trading reinforcement learning struggles with unstable, manually designed auxiliary tasks. QUESTrader automatically discovers GVF-based questions with meta-gradients, achieving consistent improvements across four equity indices, including higher returns and risk-adjusted ratios.
Problem
Stock-trading reinforcement learning needs stable representations, but manually designed auxiliary tasks depend on fixed targets and horizons that may not generalize across changing market regimes.
Method
QUESTrader uses a dual-network PPO framework that discovers auxiliary questions as GVFs and tunes their cumulants and discounts through non-myopic meta-gradients.
Results
QUESTrader achieves the highest annual and cumulative returns across DJI, FTSE, Sensex, and TAIEX, with superior Sharpe, Calmar, and Sortino ratios and competitive drawdowns.
Takeaways & Limitations
Automatically discovered auxiliary questions produce a shared representation that is richer and more stable for reinforcement-learning stock trading across the evaluated indices.
Takeaways & Limitations
The study focuses on on-policy GVFs and does not dynamically adapt the number of discovered questions or unroll length during training.
Abstract
from arXiv · showhide
Reinforcement learning has gained increasing attention as a data-driven approach for stock trading. However, learning a policy that is both profitable and stable remains challenging due to non-stationary market behaviour and noisy reward signals. Auxiliary tasks are often used to improve representation learning and stabilize training, yet they are usually designed manually and depend heavily on prior assumptions about targets and prediction horizons. Such fixed designs may not remain suitable across changing market regimes. In this work, we propose a self-supervised framework that automatically discovers auxiliary tasks to support reinforcement learning for stock trading. The auxiliary tasks are formulated as General Value Functions so that their predictions enrich the learned state representation and assist policy optimization. The framework consists of two networks. The main network learns the trading policy along with the auxiliary predictions, while the secondary network generates the definitions of auxiliary tasks through learned cumulants and discount factors. These tasks are updated using a meta gradient mechanism that accounts for their long-term impact on trading performance and improves training stability. We evaluate the proposed approach across four major equity indices: DJI, FTSE, Sensex, and TAIEX. The empirical results demonstrate that automatically discovered auxiliary tasks lead to more robust learning and improved trading performance compared to existing baselines.
1. Introduction
Automated stock trading is motivated by the complexity and scale of financial data, while reinforcement learning addresses trading as sequential decision-making but can learn inefficiently from sparse extrinsic rewards. QUESTrader addresses this challenge by automatically discovering GVF-based auxiliary tasks through a two-network architecture and non-myopic meta-gradients, achieving higher returns and superior risk-adjusted ratios with competitive drawdowns.
- Motivation: Algorithmic trading systems help process high-frequency equity-market data and identify opportunities beyond individual human traders’ cognitive capabilities.Institutional and retail traders have widely adopted systems operating with minimal human involvement.
- Motivation: Supervised price and trend prediction can succeed in stable regimes with sufficient labeled data but is limited by changing market conditions.Applied techniques include linear regression, decision trees, support vector machines, and deep neural networks.
- Reinforcement-learning formulation: In reinforcement learning, an agent repeatedly observes market states, chooses buy, sell, or hold actions, and receives profit-or-loss feedback to maximize expected cumulative rewards.The trading process is modeled as a Markov Decision Process with discrete interaction steps.
- Auxiliary tasks: A sole extrinsic reward can produce low learning efficiency when feedback is delayed or infrequent, motivating auxiliary tasks that provide additional pseudo-objectives derived from the environment.Auxiliary tasks are learned alongside the main objective to guide the agent’s learning process.
- QUESTRader: QUESTRader automatically discovers auxiliary-task questions for stock trading because handcrafted tasks may lack robustness and generalization across dynamic market conditions.The method is designed to reduce reliance on domain knowledge and improve scalability across diverse, evolving market environments.
- Contributions and evaluation: QUESTRader uses an answer network for policy and generated-question answers, a question network that formulates GVFs, and a non-myopic meta-gradient approach.Experiments report higher returns and superior risk-adjusted ratios with competitive drawdowns than strong baselines, alongside ablations of discovered-question count and inner unroll length.
2. Related Work
Related work applies reinforcement learning to stock trading through recurrent, deep, and adaptive methods, while auxiliary tasks improve market representations and performance. Existing approaches also address changing market conditions through regime selection, attention, and dynamically weighted ensembles.
- RL for Stock Trading: RL methods have become increasingly prevalent in automated stock trading, beginning with Moody et al.’s recurrent RL model optimized using differential Sharpe ratio.The model’s empirical studies demonstrated effectiveness, according to the passage.
- Adaptive Trading Methods: Efficient deep SARSA integrates bidirectional LSTM and attention for trading in volatile markets.
- Adaptive Trading Methods: DREB dynamically adapts to varying market conditions by using Bayesian optimization to assign time-varying weights to base DRL models.The approach is described as an ensemble method.
- Auxiliary Tasks: Auxiliary tasks serve as secondary objectives that improve market representation and overall performance, including volatility prediction in the risk-aware DeepScalper framework.The volatility-prediction task helps identify market risk while maximizing profit.
3. Background
The background establishes reinforcement learning and PPO foundations, introduces GVFs as flexible auxiliary predictions, and formalizes multi-stock trading as an MDP. It specifies the trading state, action space, and reward structure used throughout the paper.
- Reinforcement Learning: Reinforcement learning models sequential decision-making as an MDP with states, actions, transitions, rewards, and discounting, seeking policies that maximize cumulative discounted rewards.The policy maps states to action distributions, while value functions evaluate expected returns.
- Proximal Policy Optimization: PPO improves policy-training stability by softly constraining updates through a clipped surrogate objective, while actor and critic networks learn policies and value estimates.Clipping keeps successive policies relatively close and prevents excessive updates.
- General Value Functions: GVFs extend value functions by predicting discounted cumulative cumulant signals, enabling arbitrary environment-related questions and auxiliary tasks.Their cumulants can represent signals beyond conventional reward prediction, and meta-gradient methods can discover and refine such tasks.
- Stock Trading as an MDP: The multi-stock trading problem is formulated as an MDP in which the agent observes market states and chooses buy, sell, or hold decisions over T periods.For n stocks, each state is a (10n+1) dimensional vector, and the agent optimizes cumulative wealth through sequential actions.
- Stock Trading as an MDP: The trading action is an n-dimensional share-position vector with complete action-space dimension (2m + 1)n, while rewards combine instantaneous profit or loss with transaction costs.The reward incentivizes profitability while minimizing costs incurred after executing trades.
4. Methodology
QUESTRader automatically discovers General Value Function auxiliary tasks alongside PPO-based trading through a two-network architecture. A non-myopic meta-gradient updates the task definitions according to their long-term effect on trading performance, while auxiliary predictions improve the learned representation.
- Auxiliary-task discovery: The framework automatically discovers auxiliary questions as General Value Functions to improve representation learning and sample efficiency under shifting market regimes.The approach uses a non-myopic meta-gradient to optimize discovered questions for the trading objective.
- Two-network architecture: The two-network architecture assigns trading and auxiliary-answer learning to the Answer/Main network and task-definition generation to the Question network.The Question network emits cumulants and discounts defining on-policy GVFs, while PPO trains the trading agent.
- Question network: The Question network receives a short future state slice and produces cumulant and discount vectors, with each pair defining one GVF question.The number of vector pairs equals the number of discovered questions, dq.
- Main network: The Main network outputs the PPO policy, value function, and scalar predictions for discovered questions, optimizing trading performance and auxiliary-answer accuracy jointly.Auxiliary answers are trained using squared TD errors over a truncation horizon W, and λaux controls their contribution to the total loss.
- Meta-gradient optimization: The meta-gradient changes Question-network parameters according to how their induced GVF targets affect Main-network updates and long-term PPO performance.Bilevel optimization uses K inner updates of θ before evaluating the meta-loss and updating η; non-myopic gradients differentiate through the unrolled updates.
5. Experimental Setup
The experiments evaluate QUESTrader across four global equity indices using six risk-and-return metrics, diverse traditional and reinforcement-learning baselines, and consistent realistic trading conditions.
- Datasets: QUESTrader is evaluated on DJI, FTSE, Sensex, and TAIEX, representing markets in the US, UK, India, and Taiwan.The indices cover four prominent global stock markets.
- Datasets: Training and validation use data through December 31, 2023, while out-of-sample performance is evaluated from January 1, 2024, to March 31, 2025.This separates model development from subsequent trading evaluation.
- Evaluation Metrics: Performance is assessed with six metrics: Cumulative Return, Annual Return, Sharpe Ratio, Maximum Drawdown, Calmar Ratio, and Sortino Ratio.The metrics cover return, risk, and risk-return tradeoffs; Sharpe Ratio measures excess return per unit of total risk, while Sortino Ratio focuses on downside risk.
- Baselines: Baselines span market-index, buy-and-hold, MVO, PPO, VS-DRL, SRRS, DREB, PA-AXT, PPO-AXT, and DeepScalper methods.The comparison includes traditional finance, reinforcement learning, and auxiliary-task approaches.
- Trading Environment: All studies use an initial capital of 1,000,000, a 0.1% transaction fee on buy and sell orders, and execution at the day’s closing price.Bayesian optimization is used for hyperparameter tuning.
6. Results and Discussion
QUESTrader consistently outperforms baseline methods across four equity indices on return and risk-adjusted metrics while maintaining competitive drawdown. Visual analyses and ablations further indicate more stable trading behavior and beneficial intermediate settings for auxiliary-question count and unroll length.
- Comparative Performance: 21.785% annual return and 27.536% cumulative return make QUESTrader the strongest DJI performer, with a 1.459 Sharpe ratio.Its Calmar ratio is 2.394 and Sortino ratio is 2.143, both best among methods, while downside risk remains controlled relative to higher-return peers.
- Comparative Performance: 19.164% annual return and 24.596% cumulative return on FTSE accompany a 1.124 Sharpe ratio, 1.575 Calmar ratio, and 1.774 Sortino ratio.QUESTrader’s maximum drawdown is 12.165%; MVO has a lower 10.062% drawdown but substantially lower returns.
- Comparative Performance: QUESTrader achieves 16.727% annual return, 20.809% cumulative return, and a 1.003 Sharpe ratio on Sensex, improving over the cited baselines.Its Calmar ratio is 1.263 and Sortino ratio is 1.488, both best in the table; discovered GVF questions support stability during volatility and liquidity shifts.
- Comparative Performance: QUESTrader tops annual return, cumulative return, Sharpe, Calmar, and Sortino on DJI, FTSE, Sensex, and TAIEX, with competitive maximum drawdown.The framework is evaluated using returns, Sharpe ratio, maximum drawdown, Calmar ratio, and Sortino ratio across all four datasets.
- Visualizations and Ablation: The visual studies show higher returns within comparable volatility bands, fewer QUESTrader trading decisions than plain PPO, and Sharpe peaks near dq = 16 and K = 10.Sharpe is approximately 1.4 at dq = 16 and approximately 1.6 at K = 10; increasing dq to 32 causes a mild decline, while dq = 64 recovers near peak.
7. Conclusion and Future Work
QUESTRader automatically discovers GVF-based auxiliary questions alongside a PPO trading policy through a two-network, non-myopic meta-gradient framework. The study reports consistent improvements across four equity indices and identifies extensions to off-policy learning, implicit gradients, and online regime adaptation.
- Contributions: QUESTRader replaces manually designed side objectives with automatically discovered auxiliary questions formulated as General Value Functions.The framework learns these predictions alongside the trading policy to enrich the shared representation.
- Contributions: The Answer/Main network jointly learns the PPO policy and GVF answers, while the Question network emits per-time cumulants and discount factors.A non-myopic meta-gradient through an inner PPO-update unroll tunes the discovered questions.
- Empirical findings: Empirical results on DJI, FTSE, Sensex, and TAIEX indicate consistent improvement from the discovered auxiliary tasks.The resulting shared representation is described as rich, stable, and sensitive to market scale.
- Future work: Future work could extend the on-policy GVF framework to off-policy GVFs and implicit-gradient meta-updates.These extensions may reduce variance and memory overhead.
- Future work: Online regime detection could dynamically adapt the number of discovered questions, dq, and the unroll length, K, during training.This would allow adaptation to changing market regimes.