PaperlayerView exact revision on arXiv

Source-linked AI summary

QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE

Junjie Zhao, Chengxi Zhang, Min Qin, Peng Yang

arXiv:2409.05144v3q-fin.CPcs.AIcs.LG

TL;DR

Deep-model alpha factors are difficult to interpret in risk-sensitive markets, while formulaic-factor mining faces search challenges and PPO issues in this setting. QFR combines REINFORCE with variance controls and information-ratio reward shaping; experiments report a 3.83% improvement in correlation with asset returns over the previous state-of-the-art algorithm and stronger excess-profit ability.

  • Problem

    Deep-model alpha factors are poorly interpretable for risk-sensitive trading, while formulaic-factor mining faces search limitations and PPO's critic can produce biased, slow-learning signals under trajectory-only rewards.

    Source: §I · pp. 1–2

  • Method

    QFR trains formulaic-factor policies with REINFORCE, using deterministic transitions, a greedy-policy baseline to mitigate variance, and information ratio for reward shaping.

    Source: §I · p. 2

  • Results

    3.83% improvement in correlation with asset returns over the previous state-of-the-art algorithm, with stronger excess-profit ability and outperformance of prior RL, tree-based, and heuristic methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    QFR is presented as a promising formulaic-factor mining approach that produces steadier signals across market volatility and better performance than existing methods.

    Source: §I · p. 2, §VI · p. 15

Abstract

from arXiv · show

Alpha factor mining aims to discover investment signals from the historical financial market data, which can be used to predict asset returns and gain excess profits. Powerful deep learning methods for alpha factor mining lack interpretability, making them unacceptable in the risk-sensitive real markets. Formulaic alpha factors are preferred for their interpretability, while the search space is complex and powerful explorative methods are urged. Recently, a promising framework is proposed for generating formulaic alpha factors using deep reinforcement learning, and quickly gained research focuses from both academia and industries. This paper first argues that the originally employed policy training method, i.e., Proximal Policy Optimization (PPO), faces several important issues in the context of alpha factors mining. Herein, a novel reinforcement learning algorithm based on the well-known REINFORCE algorithm is proposed. REINFORCE employs Monte Carlo sampling to estimate the policy gradient-yielding unbiased but high variance estimates. The minimal environmental variability inherent in the underlying state transition function, which adheres to the Dirac distribution, can help alleviate this high variance issue, making REINFORCE algorithm more appropriate than PPO. A new dedicated baseline is designed to theoretically reduce the commonly suffered high variance of REINFORCE. Moreover, the information ratio is introduced as a reward shaping mechanism to encourage the generation of steady alpha factors that can better adapt to changes in market volatility. Evaluations on real assets data indicate the proposed algorithm boosts correlation with returns by 3.83\%, and a stronger ability to obtain excess returns compared to the latest alpha factors mining methods, which meets the theoretical results well.

I. INTRODUCTION

QFR addresses the difficulty of using opaque deep-model factors in risk-sensitive trading and the search challenges of interpretable formulaic factors. It replaces PPO-based training with REINFORCE and variance-control components, with experiments reporting improved factor performance.

I. INTRODUCTION

  • Deep-model alpha factors are poorly interpretable for risk-sensitive trading, while formulaic factors are favored but existing search methods face performance or exploration limits.Tree-based methods may bottleneck on nonlinear, high-dimensional data; genetic programming can explore broader expressions but struggles with large search spaces and is computationally expensive.Source: §I · p. 1
  • For the framework in, PPO's critic can produce biased value estimates from trajectory-only rewards, slow convergence, and add a second large network to update.Monte Carlo policy-gradient estimates use actual rewards and are unbiased, but their reliance on complete formula episodes yields high variance.Source: §I · p. 2
  • QFR trains with REINFORCE without a critic, uses deterministic MDP transitions and a greedy-policy baseline to mitigate variance, and shapes rewards with information ratio for steadier factors.The paper reports faster convergence and steadier signals across market volatility.Source: §I · p. 2
  • 3.83% improvement in correlation with asset returns over the previous state-of-the-art algorithm, alongside stronger excess-profit ability; QFR also outperforms prior RL, tree-based, and heuristic methods.Experiments span multiple real-world asset datasets, and theoretical results align with experimental trends under distinct market volatility.Source: §I · p. 2

A. Automatic Mining for Alpha Factors

  • Automatic mining includes tree and heuristic methods, while the RL framework in generates collaborative formulaic factors and addresses limitations of independently mined factors.The broader related-work review covers automatic alpha-factor mining and REINFORCE theory.Source: §II · p. 2, A. Automatic Mining for Alpha Factors · p. 3

B. The REINFORCE algorithms

The section presents REINFORCE as a possible fit for formulaic-factor mining and establishes the factor representation and MDP used to frame the problem.

B. The REINFORCE algorithms

  • REINFORCE is generally less favored than actor-critic methods, but can suit formulaic-factor MDPs when a proper baseline reduces variance.Prior work also studies baseline choices and their variance effects, including comparison of random-response and greedy-response rewards [19].Source: B. The REINFORCE algorithms · p. 3

A. Alpha Factors for Predicting Asset Prices

  • Each alpha factor maps market features to values across assets, and a weighted factor pool predicts asset prices with weights optimized by gradient descent.The pool’s weights are learned by minimizing mean squared error between predicted and ground-truth prices.Source: A. Alpha Factors for Predicting Asset… · p. 4

B. Formulaic Alpha Factors

  • Formulaic factors have interchangeable expression, tree, and RPN forms; RPN is the unique post-order traversal of the expression tree.The token set includes operators, features, time deltas, constants, and sequence markers.Source: B. Formulaic Alpha Factors · p. 4, Figure 1 · p. 4
  • The Information Coefficient measures correlation between predicted factor values and asset prices, with higher values indicating stronger predictive power.The section denotes the average IC across trading days as IC.Source: B. Formulaic Alpha Factors · p. 4

C. MDPs for Mining Formulaic Alpha Factors

  • The MDP builds an RPN formula token by token: states are partial sequences, actions are candidate tokens, transitions are deterministic, and reward arrives only for a completed factor.Legal sequences begin with BEG and end at SEP or the maximum length; the modeled environment is the factor-generation process, not market randomness.Source: C. MDPs for Mining Formulaic Alpha… · pp. 4–5

D. Comparing REINFORCE with PPO

The section contrasts Monte Carlo REINFORCE with PPO’s actor-critic approach in an MDP where feedback arrives only after a complete factor is generated.

D. Comparing REINFORCE with PPO

  • PPO’s temporal-difference approach can reuse data from each step for training, increasing sample efficiency relative to Monte Carlo methods.The comparison notes this advantage alongside the difficulty of critic training under terminal-only feedback.Source: D. Comparing REINFORCE with PPO · p. 5
  • Because reward is available only at the final step, the critic is hard to train and PPO’s advantage estimates can be less effective and highly biased.The passage contrasts this setting with PPO’s usual advantage-based actor-critic training.Source: D. Comparing REINFORCE with PPO · p. 5
  • REINFORCE estimates policy gradients from rollout samples weighted by rewards, yielding an unbiased estimate without a critic network.The method uses Monte Carlo sampling and reward-based likelihood maximization.Source: D. Comparing REINFORCE with PPO · p. 5

IV. STEADIER RL FOR ALPHA MINING

QFR combines REINFORCE with a greedy-reward baseline and information-ratio shaping to support lower-variance training and steadier factor mining.

A. The Proposed Algorithm

  • QFR uses a greedy baseline with a variance upper-bound rationale, and its analysis reports lower variance than REINFORCE and the lowest variance for deterministic versus probabilistic transitions.The baseline compares a sampled factor’s reward with the reward of a greedily generated factor.Source: §IV · p. 5, A. The Proposed Algorithm · p. 6
  • QFR’s policy generates random and greedy token trajectories, while a combination model evaluates both factors using real asset features and shaped rewards.The reward pair is used to estimate the policy update; the pipeline is shown in Fig. 3.Source: A. The Proposed Algorithm · p. 6, Figure 3 · p. 7

B. Time-varying Reward Shaping

  • The prior reward emphasizes absolute excess returns, whereas QFR’s shaping also assesses risk-adjusted signal stability.Information Ratio measures excess returns relative to excess risk; its test threshold becomes more demanding as training progresses.Source: B. Time-varying Reward Shaping · p. 6
  • IR-based shaping discounts the final IC reward when a factor’s IR falls below the time-varying clipped test value, encouraging long-term consistency over short-term profit peaks.The clip range changes over training time, as illustrated in Fig. 4.Source: B. Time-varying Reward Shaping · pp. 6–7, Figure 4 · p. 7

C. The Theoretical Analysis

The analysis establishes unbiasedness for the gradient estimator and examines variance under deterministic and probabilistic state transitions. It also bounds QFR’s estimator variance and derives conditions under which QFR reduces variance relative to REINFORCE.

C. The Theoretical Analysis

  • QFR’s gradient estimator is unbiased for the objective’s policy gradient.The analysis also establishes a variance bound for QFR’s gradient estimator.Source: C. The Theoretical Analysis · pp. 7, 9
  • Deterministic state transitions have lower variance than probabilistic transitions, helping mitigate REINFORCE’s high variance.With deterministic transitions, sequence variance comes from policy decision-making rather than an additional random transition process.Source: C. The Theoretical Analysis · p. 8
  • Under the stated softmax parameterization, QFR’s gradient-estimator variance is bounded by 8 × rmax^2 × T^2 × S^2/N, with c a universal constant.Here, S bounds the norm of the policy log-gradient and rmax is the maximum reward magnitude.Source: C. The Theoretical Analysis · p. 8
  • In a two-armed bandit with positive rewards, QFR has lower variance under the stated probability condition; if r1 < 2r2, this holds for any action probability in (0, 1).The result extends to larger action spaces when the greedy reward exceeds half the sampled reward; the paper relates this condition to industry factor ICs rarely exceeding 0.1.Source: C. The Theoretical Analysis · pp. 9–10

V. NUMERICAL RESULTS

The numerical study evaluates QFR against state-of-the-art RL algorithms and other factor-mining methods on six stock datasets. It also examines reward-shaping hyperparameters and ablates the two proposed improvements.

V. NUMERICAL RESULTS

  • The study compares QFR with state-of-the-art RL algorithms and other factor-mining methods across six stock datasets.Its evaluation includes RL efficiency, reward-shaping hyperparameters, five factor-mining methods, and an ablation of the two improvements.Source: §V · p. 10
  • The evaluation includes a separate study of reward-shaping hyperparameters.This is one component of the numerical study, alongside RL efficiency comparisons and factor-mining evaluations.Source: §V · p. 10
  • An ablation study examines the importance of the two proposed improvements.Source: §V · p. 10

A. Environment Settings

The experiments use six Chinese and US stock indices, public price-volume features, and five-day asset returns as the target. They compare QFR with tree, heuristic, deep-model, and interpretable RL baselines.

A. Environment Settings

  • The experiments use six indices—CSI300, CSI500, CSI1000, SPX, DJI, and NDX—with six public price-volume features to predict five-day asset returns.The train, validation, and test periods are 2016/01/01–2020/01/01, 2020/01/01–2021/01/01, and 2021/01/01–2024/01/01, respectively.Source: A. Environment Settings · p. 10
  • Baseline methods span tree models, heuristic search, end-to-end deep models, and interpretable reinforcement learning.The comparison follows open-source implementations of AlphaGen, gplearn, Stable Baseline 3, and Qlib.Source: A. Environment Settings · pp. 10–11
  • The RL comparison includes AlphaGen, TRPO, PPO, and A3C for the formulaic-alpha mining task.Experiments use five random seeds.Source: A. Environment Settings · p. 11

B. Comparisons with Other RLs

QFR is compared with other RL methods using Rank IC on six indices. It generally converges faster and more stably, improving the metric by 3.83% over PPO on five indices, while matching PPO on CSI500.

B. Comparisons with Other RLs

  • Rank IC, rather than reward, is used to compare learning curves because QFR uses reward shaping and the other algorithms do not.Higher Rank IC is better.Source: B. Comparisons with Other RLs · p. 11
  • QFR improves by 3.83% on the metric versus AlphaGen’s best-performing PPO across the remaining five index constituents, while QFR and PPO show similar performance on CSI500 constituents.Source: B. Comparisons with Other RLs · p. 12
  • QFR outperforms the other algorithms in convergence speed and stability on the five indices where it improves over PPO.It catches up with and surpasses initially stronger PPO/TRPO; the paper attributes the early advantage to controlled policy-update steps and the later gains to more accurate gradient estimates without that restriction.Source: B. Comparisons with Other RLs · p. 12

C. Sensitivity Analysis of the Time-varying Reward Shaping

The sensitivity analysis tunes reward-shaping parameters to balance IC and IR behavior. Excessive shaping or premature IR tests can hurt performance or destabilize training.

C. Sensitivity Analysis of the Time-varying Reward Shaping

D. Factors Evaluation

QFR performs better than the compared factor-mining methods in the reported evaluation, while the authors attribute baseline differences to data and optimization behavior.

D. Factors Evaluation

E. Investment Simulation

In CSI300 investment simulations, QFR generally outperforms baseline strategies across time horizons and selected volatility conditions. The evaluation uses a daily top-50 trading strategy.

E. Investment Simulation

  • The testing-phase simulation uses CSI300 constituents from 2021/01/01 to 2024/01/01, holding the 50 stocks ranked highest by mined factor values and rebalancing when membership changes.Source: E. Investment Simulation · pp. 12–13
  • Table V compares quarterly market conditions and risk metrics for CSI300 investment strategies driven by QFR and AlphaGen.Source: Table V · p. 13
  • QFR outperforms all baselines on weekly, quarterly, and yearly cumulative returns, but slightly trails AlphaGen on monthly returns.Fig. 7 tracks cumulative returns, and Table VI reports results across time horizons.Source: E. Investment Simulation · p. 13, Table VI · p. 13, Figure 7 · p. 13
  • QFR achieves the most profit across selected high-, changing-, and low-volatility periods, with a significant advantage during high volatility.The periods include CIMV above 20% in 2021, a 7.43% CIMV decrease in 2022, and CIMV below 20% in 2023.Source: E. Investment Simulation · p. 13, Figure 8 · p. 14

F. Ablation Study

The ablation study finds that the baseline and reward shaping work best together. The conclusion presents QFR as a strong formulaic-factor mining method and identifies possible extensions.

F. Ablation Study

  • Removing the baseline degrades factor quality, while removing reward shaping gives faster early learning but ultimately falls to local optima.The authors suggest the unshaped variant may overemphasize absolute returns.Source: F. Ablation Study · p. 14
  • Using all improvements gives the best performance, indicating that the baseline and reward shaping are complementary.Source: F. Ablation Study · p. 15

VI. CONCLUSION

  • QFR achieves better performance than state-of-the-art RL algorithms and generates superior factors compared with existing mining methods on real-world asset data.Source: §VI · p. 15
  • Future directions include matrix tri-factorization for cross-stock correlations and community detection for latent asset clusters and investor behavior patterns.The conclusion also notes potential extensions to formula-based signal-processing tasks.Source: §VI · p. 15
Loading 2409.05144v3…