AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning

1 Introduction

• Existing RL methods face non-stationarity because rewards depend on the evolving alpha pool, which their state representations generally omit.The same state-action trajectories can therefore receive different rewards as the pool changes.Source: §1 · p. 2
• Existing methods optimize a single scalar, typically mega-alpha IC, despite pool quality also depending on temporal stability, perturbation robustness, and diversity.High IC alone may coexist with temporal fluctuation or fragility under market-regime shifts.Source: §1 · p. 2
• AlphaPareto augments the state with the evolving pool, uses an LLM to encode its semantic content, and optimizes four reward dimensions through Pareto-regularized learning.The objectives are predictive power, temporal stability, perturbation robustness, and diversity; the approach seeks policies along the efficient frontier.Source: §1 · p. 2
• The paper frames its conceptual contribution as reformulating alpha discovery into a context-conditional stationary MDP with a multi-objective reward.The authors describe this reformulation as better aligned with practical challenges.Source: §1 · p. 2
• Experiments on large-scale real-world Chinese stock-market datasets show AlphaPareto consistently outperforms state-of-the-art baselines.Ablation studies also confirm complementary contributions from its proposed components.Source: §1 · p. 2

GP-based alpha discovery

• Genetic programming evolves expression trees using IC, but scales poorly with more operators and features and does not directly optimize alpha-pool synergy.The paper associates this lack of synergy optimization with poor practical performance.Source: §2 · p. 3

RL-based alpha discovery

• RL-based methods model formulaic alpha discovery as sequential generation, but share limitations in pool analysis and single-scalar reward optimization.AlphaGen uses MaskPPO; AlphaQCM adds an exploration bonus to reduce non-stationarity's impact, while AlphaForge incorporates temporal performance into selection.Source: §2 · p. 3

LLM-based alpha discovery

• Most cited LLM-based methods use LLMs to generate factors directly, whereas AlphaPareto uses one to semantically encode the alpha pool within an RL loop.Examples include AlphaAgent's market-hypothesis proposals and R&D-Agent-Quant's Research and Development stages.Source: §2 · p. 3

3 Problem Setup

• Formulaic alphas map historical market data to cross-sectional signals, which a linear model combines into a mega-alpha.The stated goal is a pool of P formulaic alphas with strong out-of-sample predictive power.Source: §3 · p. 3
• Alphas are represented as reverse Polish notation token sequences, and sequential token selection is modeled as an MDP.Invalid-action masking restricts choices to keep sequences valid; episodes terminate at SEP or when the maximum sequence length is exceeded.Source: §3 · pp. 3–4
• A valid candidate is temporarily added to the pool, a linear model is fitted, and the pool retains at most P alphas based on fitted contributions.The reward is the updated pool's in-sample IC; incomplete candidates receive zero reward and invalid ones receive −1.Source: §3 · p. 4
• Because reward depends on the evolving pool, identical state-action trajectories can receive different rewards; including pool context is difficult because its expressions are complex.This pool-dependent reward makes the discovery process inherently non-stationary.Source: §3 · p. 4
• The single-objective reward measures only IC, while pool quality also includes temporal stability, perturbation robustness, and diversity.The passage notes that high IC may coexist with temporal variation or fragility under market-regime shifts, affecting out-of-sample predictive performance.Source: §3 · p. 4

4.1 Alpha Pool Semantic Reasoning

• The state pairs formula tokens with the current pool; a frozen LLM summarizes pool composition while an LSTM encodes tokens, and their joint representation conditions the policy and value heads.The structured pool prompt is designed to capture composition, concentration, and potential blind spots.Source: §4.1 · pp. 4–5

4.2.1 Multi-objective Reward

• The method measures the four-dimensional contribution of f_t to the current alpha pool through predictive power, temporal stability, perturbation robustness, and diversity.Source: §4.2.1 · p. 5
• PFS scores robustness by correlating mega-alpha rankings before and after perturbation, using Gaussian noise for microstructure fluctuations and heavy-tailed t3 noise for structural shocks.The two noise distributions are rescaled to match empirical cross-sectional variance of stock returns.Source: §4.2.1 · p. 6

4.2.2 MAP-based Optimization

• MAP uses minimum target levels for each objective as a value palette, while adaptively balancing them with weights recomputed for online alpha discovery.The authors choose MAP because its targets remain in each objective's original units, unlike fixed weights whose meaning changes with objective scale.Source: §4.2.2 · pp. 6–7

5 Experiments

• Experiments used Chinese A-share data with a 20-day future-return target across CSI300, CSI800, and the full Market universe, with chronological train, validation, and test periods.The study compares handcrafted and non-formulaic baselines, GP, LLM-based generators, and RL-based discovery methods; stochastic experiments use five random seeds.Source: §5 · p. 7

5.1 Main Comparison

• AlphaPareto recorded out-of-sample ICs of 3.92% (CSI300), 5.70% (CSI800), and 10.10% (Market), the highest across methods.The second-best methods achieved 3.69%, 5.07%, and 9.16%, respectively.Source: Table 1 · p. 8, §5.1 · p. 8
• On CSI300 and CSI800, AlphaPareto ranked highest in perturbation robustness, second in diversity, and third or fourth in temporal stability; on Market, robustness led while diversity and stability were moderate.Figure 2 compares these three out-of-sample dimensions.Source: §5.1 · p. 8, Figure 2 · p. 9
• AlphaPareto achieved the highest annualized return (58.92%), information ratio (2.62), and Sharpe ratio (2.59), while its −18.12% maximum drawdown trailed GP and AlphaGen.The authors characterize its overall portfolio-level performance as strongest among evaluated methods.Source: §5.1 · p. 8, Table 2 · p. 9

5.2 Scaling Performance

• Across LLM scales, AlphaPareto performed strongly but non-monotonically, and Qwen-Embedding-4B achieved the best ICs on all datasets.The authors suggest a small or medium frozen semantic encoder may provide sufficient pool information, making the scaling results more cost-effective for deployment.Source: §5.2 · p. 9, Table 3 · p. 9

5.3 Ablation Study

• LLM-based pool reasoning mainly improved Market results, MAP-based MORL improved all three datasets, and the combined method led on CSI800 and Market while staying near the best CSI300 mean.On CSI300, the combined method had a 0.07% mean loss and 0.81% lower standard deviation; the authors hypothesize that LLM domain noise may contribute to the decline.Source: §5.3 · p. 10, Table 4 · p. 10

6 Conclusion

• Experiments on Chinese and U.S. data show AlphaPareto outperforming existing baselines across multiple dimensions; scaling and ablation studies indicate its performance does not require the largest LLMs and its components play complementary roles.The conclusion also identifies prompt-template dependence and the linear mega-alpha combiner as limitations.Source: §6 · p. 10

A Reverse Polish Notation of Formulaic Alpha

• Formulaic alphas are represented as RPN token sequences, with features and operators as tokens and BEG and SEP marking the expression’s beginning and end.Figure 3 shows an example alpha, its expression tree, and its RPN representation.Source: A Reverse Polish Notation of Formulaic… · p. 12, Figure 3 · p. 12

B Prompt Template

• A structured prompt asks a frozen LLM to summarize the evolving pool’s expressions, weights, information coverage, and potential blind spots.The prompt uses a JSON file containing the pool’s expressions and fitted weights.Source: B Prompt Template · p. 12

C Algorithm

• Algorithm 1 presents pseudocode for the AlphaPareto method.Source: C Algorithm · p. 14

D Details About Baseline Methods

• Unless otherwise noted, formulaic-alpha discovery baselines use the same downstream linear mega-alpha construction, focusing comparison on selected pool quality rather than combiner complexity.Source: D Details About Baseline Methods · p. 14
• Alpha158 uses 158 predefined Qlib alphas combined by a linear model, while MLP directly predicts future returns as a non-formulaic end-to-end model.This pair represents a handcrafted pool and a black-box predictor.Source: D Details About Baseline Methods · p. 14
• The GP baseline evolves expressions individually using IC as fitness and builds its final pool from the top-P expressions, rather than directly optimizing pool-level synergy during generation.Source: D Details About Baseline Methods · p. 14
• For fair comparisons, AlphaAgent generates 10 candidates per iteration over 5 iterations, while R&D-Agent-Quant uses 10 factors per iteration over 10 loops and the shared structured expression syntax.Both LLM baselines use the LLM to generate candidates rather than as a frozen semantic encoder inside an RL loop.Source: D Details About Baseline Methods · pp. 14–15
• AlphaGen, AlphaForge, and AlphaQCM are RL-based competitors, and their comparison with AlphaPareto is intended to isolate pool-aware state augmentation and Pareto-regularized multi-objective optimization.The passage characterizes the competitors as optimizing scalar rewards tied primarily to predictive performance without explicitly encoding current-pool semantics.Source: D Details About Baseline Methods · pp. 15–16

E.1 Network Hyperparameters

• The method uses a two-layer LSTM with hidden dimension 128 and dropout 0.1; FFNe and FFNh have hidden dimension 256, and FFNu maps the fused representation to 128 dimensions.The LLM embedding dimension depends on the selected model; Qwen-Embedding-4B is given as an example with dimension 2560.Source: §E.1 · p. 16
• Table 6 reports embedding dimensions for LLM candidates.Source: Table 6 · p. 16

E.2 RL Hyperparameters

• Additional RL hyperparameters are listed in Table 7.Source: §E.2 · p. 16, Table 7 · p. 16

F Paired Significance Tests

• The comparisons use paired t-tests across five matched random seeds, with a positive t-statistic favoring AlphaPareto.Source: F Paired Significance Tests · p. 17
• At the 10% significance level, AlphaPareto has significantly higher mean IC than every baseline on CSI300 and CSI800, and than all but AlphaQCM on Market.The Market comparison with AlphaQCM has a positive estimated difference but is not significant (t = 0.65, p = 27.60%).Source: Table 8 · p. 17, F Paired Significance Tests · p. 17

• The backtest selects the top 50 stocks by mega-alpha, partially rotates holdings by up to five eligible stocks per rebalance, and excludes stocks under price limits.Trades execute at the close; the portfolio is long-only and rebalanced monthly.Source: G Additional Details and Experiments… · p. 17
• Transaction costs are 15 basis points per side with a 5 CNY minimum fee, and the Shanghai Composite Index is the benchmark.The backtest uses an initial capital of 100 million CNY and the full market universe.Source: G Additional Details and Experiments… · p. 17
• AlphaPareto delivers the strongest cumulative return over the test period while maintaining a controlled drawdown profile.Figure 4 compares cumulative returns of monthly portfolios constructed by different methods.Source: G Additional Details and Experiments… · p. 17, Figure 4 · p. 17

H Impact of MORL method

• MAP expresses interpretable minimum target levels in each objective's original units, while deriving nonnegative scalarization weights from the observed reward distribution.The authors note that this retains subjective preferences rather than eliminating them.Source: H Impact of MORL method · p. 18
• On CSI800, MAP-based AlphaPareto achieves a mean IC of 5.70% (standard deviation 0.87%), versus 4.92% (0.52%) for equal-weight scalarization.The comparison supports MAP's benefit over this fixed-weight alternative, although MAP's standard deviation is higher; more sophisticated MORL optimizers may yield further gains.Source: H Impact of MORL method · p. 18

I Random-Noise Control

• On CSI800, matched random noise yields mean out-of-sample IC of 4.22% (1.62% standard deviation), versus 5.70% (0.87%) for AlphaPareto and 5.54% (1.03%) without pool representation.This control is consistent with useful pool-dependent semantic information rather than matched pool-independent noise alone.Source: I Random-Noise Control · p. 18, Table 9 · p. 18

J Empirical Results on the S&P 500 Universe

• On S&P 500, AlphaPareto improves mean IC over AlphaQCM from 7.18% to 7.65% and reduces standard deviation from 0.84% to 0.13%.It has the highest mean IC among evaluated methods and the lowest standard deviation among methods with repeated runs; the evaluation changes only the stock universe.Source: J Empirical Results on the S&P 500… · pp. 18–19, Table 10 · p. 19

K Computational Resources

• Each AlphaPareto training run used one RTX PRO 6000 GPU and 64 CPU workers; stochastic experiments were repeated over five random seeds.The reported experiments ran on servers with 66 Intel Xeon Platinum 8470Q vCPUs and 330 GB RAM.Source: K Computational Resources · p. 19, Table 11 · p. 19