PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

Arishi Orra, Himanshu Choudhary, Manoj Thakur

arXiv:2608.15770v1cs.LGstat.ML

TL;DR

Stock-trading inverse reinforcement learning is limited by heterogeneous expert behavior, unstable exploration, and explicit risk-control challenges. BRaG combines barycenter-based multi-expert imitation learning with reinforcement learning and control barrier functions, outperforming classical and deep reinforcement learning methods across four markets while maintaining more stable risk characteristics.

  • Problem

    Existing stock-trading inverse reinforcement learning methods often use narrowly defined experts, face unstable exploration, and lack explicit risk constraints across evolving market regimes.

  • Method

    BRaG aggregates heterogeneous expert demonstrations with a performance-weighted Wasserstein barycenter, pretrains via adversarial imitation learning, and applies risk-constrained reinforcement learning.

  • Results

    Across four global equity markets, BRaG achieves the strongest returns, highest Sharpe ratios, lower maximum drawdowns, and more stable portfolio evolution than competing methods.

  • Takeaways & Limitations

    BRaG provides a multi-market stock-trading policy with stronger returns and comparatively more stable risk characteristics than classical rules and recent DRL baselines.

Abstract

from arXiv · show

Designing effective trading strategies using reinforcement learning remains challenging due to delayed and noisy rewards, poor exploration, and the difficulty of enforcing explicit risk constraints. In this work, we propose BRaG, a barycenter-based adversarial inverse reinforcement learning framework for stock trading that learns trading behavior from multiple heterogeneous expert strategies. BRaG aggregates expert demonstrations using a performance-weighted Wasserstein barycenter, yielding a stable pseudo-expert representation that captures shared structure across diverse trading styles. This representation is used to pretrain a trading policy via adversarial imitation learning, which alleviates unstable exploration during reinforcement learning. The pretrained policy is subsequently refined using reinforcement learning with true market rewards. To ensure risk-aware decision-making, BRaG incorporates control barrier functions that constrain action execution and regularize policy learning to satisfy drawdown limits. We evaluate the proposed approach on four major global equity markets, including the US, UK, Indian, and Taiwanese indices. Across all the markets, the proposed approach achieves stronger performance than both classical trading rules and recent deep reinforcement learning methods, while exhibiting more stable risk characteristics.

1 INTRODUCTION

The introduction identifies delayed, noisy rewards, heterogeneous expert behavior, and weak exploration as key obstacles to stock-trading reinforcement learning. It presents BRaG, which combines performance-weighted Wasserstein barycenters, adversarial imitation pretraining, and control barrier functions for risk-aware trading.

  • Motivation: Stock trading is difficult because prices are volatile, non-stationary, and sensitive to exogenous events, challenging consistent decision-making.These market characteristics motivate automated trading methods that can adapt to changing conditions.
  • Challenges: DRL faces sparse, delayed, and noisy market feedback, while short-term profit rewards can produce unstable learning and encode implicit risk assumptions.The introduction emphasizes reward design over extended horizons as a central deployment challenge.
  • Challenges: Existing IRL methods often use single-expert or narrowly defined demonstrations, risking overfitting and poor generalization across heterogeneous, evolving market regimes.Adversarial imitation learning is also commonly used without explicitly addressing exploration challenges.
  • BRaG framework: BRaG aggregates heterogeneous expert strategies with a performance-weighted Wasserstein barycenter to construct a stable pseudo-expert representation for imitation learning.The barycenter reduces sensitivity to individual expert bias and improves robustness across diverse market conditions.
  • BRaG framework: BRaG uses adversarial imitation learning to pretrain the trading policy, initializing expert-aligned behavior to mitigate poor exploration in reinforcement learning.This pretraining mechanism is introduced specifically for financial-market reinforcement learning.
  • BRaG framework: BRaG integrates control barrier functions for explicit risk control by enforcing drawdown constraints during trading-policy learning.The supplied contribution passage introduces this risk-control component but is truncated after stating the drawdown constraint.

2 PRELIMINARIES

The preliminaries frame stock trading as an MDP optimized with policy-gradient actor–critic methods, introduce IRL for inferring latent rewards from expert demonstrations, and describe adversarial imitation learning as trajectory distribution matching.

  • Reinforcement Learning: Stock trading is modeled as a Markov Decision Process in which an agent observes market states, takes trading actions, and receives portfolio-performance rewards.The objective is to maximize expected cumulative return over time.
  • Reinforcement Learning: The framework adopts policy-gradient actor–critic optimization using Proximal Policy Optimization (PPO) for stable learning.
  • Inverse Reinforcement Learning: Inverse reinforcement learning infers an underlying reward function and latent preferences from expert demonstrations rather than relying on manually specified objectives.This is useful in finance because market feedback is noisy, delayed, and non-stationary.
  • Adversarial Imitation Learning: Generative Adversarial Imitation Learning (GAIL) formulates policy learning as matching expert and agent trajectory distributions.A discriminator distinguishes expert samples from policy-generated samples while the policy learns behavior that is indistinguishable from expert demonstrations.

3 PROBLEM FORMULATION

The paper formulates multi-stock trading as a sequential decision problem modeled by an MDP, where agents dynamically update portfolios using delayed, path-dependent profit and risk feedback. The formulation specifies market-and-portfolio states, discrete multi-stock actions, and rewards based on net portfolio-value changes.

  • MDP formulation: Trading is modeled as an MDP because portfolio decisions are sequential, while rewards depend on delayed and path-dependent profit and risk feedback.This formulation enables reinforcement learning to optimize trading strategies dynamically over time.
  • State: For n assets, the state includes cash, share holdings, closing prices, and eight technical indicators per stock, giving dimension 10n + 1.The state combines market features with portfolio information.
  • Action: Each action is an n-dimensional share-trading vector, with components from −m to m representing purchases, sales, or holding decisions.The parameter m limits shares traded per stock in one period, producing an action-space cardinality of (2m + 1)n.
  • Reward: The reward is the percentage change in marked-to-market portfolio value after trading, net of transaction costs and a fixed commission fee.This objective promotes profit-maximizing trades while penalizing excessive transactions to balance returns and cost efficiency.

4 METHODOLOGY

BRaG learns a risk-aware trading policy from heterogeneous experts by constructing a performance-weighted sliced Wasserstein barycenter of expert state-action behaviors, then applying adversarial imitation learning and PPO. It enforces maximum-drawdown safety with control barrier functions and refines the pretrained policy using true environment rewards.

  • Barycenter-based expert aggregation: The barycenter preserves shared behavioral structure across experts rather than selecting one expert or indiscriminately pooling demonstrations.The method uses discounted occupancy measures to represent long-term expert behavior and seeks a consensus representation across distinct trading strategies.
  • Barycenter-based expert aggregation: BRaG aggregates heterogeneous expert trajectories into a consensus pseudo-expert using a performance-weighted sliced Wasserstein barycenter.Expert weights are proportional to non-negative Sharpe ratios, excluding experts with negative risk-adjusted performance; random projections and quantile averaging produce pseudo-expert samples.
  • Adversarial imitation learning: GAIL trains a policy and discriminator adversarially, using discriminator-derived rewards to align policy occupancy with barycenter expert behavior.BRaG avoids explicit reward parameterization, and PPO provides stable policy updates through clipped likelihood ratios and value-function regularization.
  • Risk-aware control: Control barrier functions enforce a maximum drawdown constraint by projecting unsafe actions onto the safe set during execution.The barrier is defined over portfolio value and its running maximum, with safety ensured when B(V_t) ≥ 0.
  • Staged policy training: BRaG uses staged training: adversarial imitation learning first pretrains the policy from barycenter demonstrations, then reinforcement learning fine-tunes it with the true environment reward.CBF constraints are applied during rollouts, while PPO updates the policy in both stages.

5 EXPERIMENTAL EVALUATION

BRaG is evaluated on four global equity indices against classical, technical, and deep reinforcement learning baselines under consistent daily trading settings. Across markets, it achieves stronger returns, risk-adjusted performance, and portfolio stability while limiting drawdowns.

  • Datasets: The evaluation uses daily stock price data from the DJI, FTSE 100, BSE Sensex, and TAIEX/TWII indices.The experiments represent the United States, United Kingdom, India, and Taiwan.
  • Baselines: BRaG is compared with passive benchmarks, classical optimization, technical trading rules, and deep reinforcement learning models.Each DRL-based model is trained over five independent runs, with average evaluation metrics reported.
  • Performance results: BRaG achieves the strongest overall cumulative and annualized returns across all four evaluated markets.The method consistently outperforms the strongest competing methods on the DJI, FTSE, Sensex, and TWII datasets.
  • Risk-adjusted results: BRaG achieves the highest Sharpe ratio across all datasets while maintaining comparatively lower maximum drawdowns.Several DRL baselines show higher return variability despite competitive profitability.
  • Stability and robustness: BRaG achieves higher win ratios across all markets, indicating that its gains reflect consistent decision-making rather than a few extreme outcomes.Its out-of-sample portfolio trajectories are generally upward with comparatively controlled declines during volatile intervals.
  • Mechanisms: BRaG’s performance benefits from diverse-expert aggregation, adversarial pretraining, and explicit risk control.These components provide a structured learning starting point, reduce unstable exploration, and limit drawdowns during market volatility.

6 CONCLUSION

BRaG combines multi-expert imitation learning, adversarial training, barycenter-based aggregation, and explicit risk control for stock trading. Its adversarial pretraining initializes policy learning meaningfully and reduces unstable exploration during reinforcement learning.

  • Conclusion: BRaG combines multi-expert imitation learning with adversarial training and explicit risk control for stock trading.The framework is presented as integrating these components into a unified trading approach.
  • Conclusion: Barycenter aggregation of heterogeneous expert strategies produces a stable reference behavior that guides policy learning.The barycenter construction is used to combine diverse expert strategies into a reference for the policy.
  • Conclusion: Adversarial pretraining initializes the policy in a meaningful region of the policy space and reduces unstable exploration during reinforcement learning.This pretraining stage is described as improving the starting point for subsequent reinforcement learning.

A PROXIMAL POLICY OPTIMIZATION · B EXPERT TRAJECTORIES

PPO provides stable, efficient policy updates through constrained clipped objectives, while four rule-based technical strategies generate diverse expert trajectories spanning momentum, trend-following, and mean-reversion behaviors.

  • A PROXIMAL POLICY OPTIMIZATION: PPO is a first-order actor-critic policy-gradient method that constrains updates near the previous policy to mitigate instability from large changes.It optimizes πθ(a|s) using stochastic gradient ascent while keeping the new policy close to πθold.
  • A PROXIMAL POLICY OPTIMIZATION: PPO maximizes a clipped surrogate objective based on the probability ratio between new and old policies and an estimated advantage function.The clipping hyperparameter ϵ controls the trust-region size and limits samples where the new policy deviates substantially.
  • A PROXIMAL POLICY OPTIMIZATION: PPO combines a learned value function with entropy regularization to reduce variance and encourage exploration.These components complement clipping, which prevents excessively large policy updates.
  • B EXPERT TRAJECTORIES: The expert set uses rule-based technical trading strategies designed to capture distinct market behaviors and complementary perspectives on trend persistence and price deviations.The demonstrations include momentum, trend-following, and mean-reversion strategies.
  • B EXPERT TRAJECTORIES: TSMOM takes long positions when an asset’s cumulative past return is positive and short positions otherwise, capturing persistent individual-asset trends.Its lookback window has length L, and the strategy adapts to regimes where trends strengthen or weaken.
  • B EXPERT TRAJECTORIES: CSMOM ranks assets by past returns, assigning long positions to top-ranked assets and short positions to bottom-ranked assets.Its long-short construction exploits relative performance differences and provides diversification benefits.
  • B EXPERT TRAJECTORIES: Moving average crossover generates trend-following signals from the interaction of short-term and long-term price moving averages.The strategy filters short-term noise while responding to sustained price movements.
  • B EXPERT TRAJECTORIES: Bollinger Bands use a moving average and volatility-based envelope to identify significant price deviations associated with mean reversion.The strategy captures overbought and oversold conditions and complements momentum experts by focusing on price corrections.

C BASELINE TRADING STRATEGIES

BRaG is evaluated against a broad set of trading baselines spanning rule-based technical experts, standard deep reinforcement learning agents, traditional portfolio strategies, and recent literature methods.

  • Rule-Based Technical Experts: Rule-based technical baselines include CSMOM, TSMOM, MA Crossover, and Bollinger Band strategies.
  • Standard DRL Agents: Standard DRL baselines comprise vanilla A2C, DDPG, and PPO algorithms.
  • Traditional Finance Baselines: Traditional finance baselines include the Market Index, Buy-and-Hold, Random Trading, and Mean-Variance Optimization.The Market Index measures broader market performance; Buy-and-Hold assumes no rebalancing, Random Trading provides a statistical floor, and MVO balances expected return against risk.
  • State-of-the-Art Literature: State-of-the-art comparisons include Adaptive, VS-DRL, SRRS, DREB, and RSHF methods.These methods use actor-critic ensembles, volatility-scaled rewards, Sharpe-ratio reward shaping, Bayesian ensemble weighting, or human-feedback reward shaping.

D PERFORMANCE METRICS

The study evaluates trading strategies with standard metrics covering profitability, risk-adjusted return, downside risk, and trade outcomes. These include cumulative and annualized returns, Sharpe ratio, maximum drawdown, and win ratio.

  • Cumulative Return measures the net change in portfolio value between the beginning and end of the trading period.
  • Annualized Return represents the average yearly growth rate of the portfolio.
  • Sharpe Ratio compares mean excess return with return standard deviation to quantify return per unit of risk.
  • Maximum Drawdown measures the largest observed portfolio decline from a historical peak to a subsequent trough.
  • Win Ratio is the percentage of executed trades yielding positive outcomes relative to total trades.

E HYPERPARAMETER SETTING

Experiments use uniform training and evaluation settings for fair model comparison. BRaG hyperparameters are selected through Bayesian optimization on a held-out validation split using Hyperopt’s tree-structured Parzen estimators.

  • Experiments use uniform training and evaluation settings to ensure fair comparison across models.
  • BRaG hyperparameters are tuned with Bayesian optimization on a held-out validation split derived from the training data.The approach is intended to efficiently search high-dimensional parameter spaces.
  • The tuning process uses Hyperopt, which implements tree-structured Parzen estimators to guide optimization.

F ADDITIONAL RESULTS

Additional results show that BRaG delivers steadier cumulative returns and robust performance across markets, while achieving stronger downside-risk and risk-adjusted characteristics than the compared strategies. These findings support the role of expert-guided pretraining and explicit risk constraints in producing more controlled trading behavior.

  • Cumulative returns: BRaG maintains a relatively steady upward trajectory across all four markets, with fewer abrupt declines than most learning-based baselines.Several DRL methods show rapid growth followed by reversals during volatile intervals, whereas BRaG appears to moderate exposure.
  • Performance robustness: BRaG achieves the highest mean performance while exhibiting greater robustness than standard DRL models across five independent random seeds.A2C, DDPG, and PPO demonstrate moderate to high performance variance across initializations; deterministic baselines have zero variance.
  • Risk characteristics: The results support combining expert-guided pretraining with explicit risk constraints to produce more controlled trading behavior.Calmar and Sortino distributions complement return-based measures by examining downside risk, capital preservation, and risk-adjusted performance.
  • Risk characteristics: BRaG consistently attains higher Calmar values across the DJI, FTSE, Sensex, and TWII datasets, indicating gains without excessive maximum-drawdown risk.Several DRL baselines achieve reasonable returns but lower Calmar ratios, while rule-based and random strategies are weaker in downside-adjusted performance.

G ABLATION STUDY

The DJI ablation study evaluates how expert aggregation, drawdown constraints, expert weighting, component removal, and demonstration aggregation affect BRaG’s cumulative return and Sharpe ratio. Results indicate that incorporating more experts and appropriately calibrated risk control improve the framework’s performance and stability.

  • Expert trajectory ablation: Adding more expert trajectories produces clear cumulative-return improvements, with experts selected incrementally by validation performance.The study varies the number of trajectories from one to five, where the single-expert setting uses the best-performing expert and the final setting uses all five.
  • Drawdown constraint ablation: Intermediate drawdown constraints provide the best balance between cumulative return and Sharpe ratio, whereas very tight or loose limits are suboptimal.Removing the constraint yields high cumulative returns but weaker risk-adjusted performance; the control barrier function stabilizes learning when appropriately calibrated.
  • Expert weighting ablation: Sharpe-based expert weighting assigns greater barycenter influence to experts with stronger risk-adjusted validation performance than equal weighting.The comparison tests the weighting scheme used by BRaG against equal weighting, where every expert contributes identically.
  • Component-wise ablation: Component-wise ablation removes GAIL pretraining, PPO fine-tuning, or the control barrier function individually to measure each module’s contribution to BRaG.Each variant retains the remaining components, isolating the effects of the three main mechanisms on the DJI dataset.
  • Demonstration aggregation ablation: Demonstration aggregation is compared across a best single expert, uniform sampling, Sharpe-weighted sampling without a barycenter, and the full BRaG method.This comparison separates the value of using multiple experts from the contribution of the specific barycenter construction technique.
Loading 2608.15770v1…