PaperlayerView exact revision on arXiv

Source-linked AI summary

FactorMiner: A Self-Evolving Agent with Skills and Experience Memory for Financial Alpha Discovery

Yanlong Wang, Jian Xu, Hongkang Zhang, Shao-Lun Huang, Danny Dongning Sun, Xiao-Ping Zhang

arXiv:2602.14670v2q-fin.TRcs.MA

TL;DR

Formulaic alpha mining must search a vast expression space while preserving interpretable signals and avoiding redundancy as factor libraries grow. FactorMiner combines a modular executable mining skill with structured experience memory in a retrieve-generate-evaluate-distill loop. Across datasets and markets, it builds a diverse, competitive factor library while maintaining low redundancy under the Correlation Red Sea constraint.

  • Problem

    Formulaic alpha discovery faces a vast combinatorial search space, limited reuse of knowledge across sessions, interpretability requirements, and shrinking feasible regions as factor libraries grow.

  • Method

    FactorMiner combines a modular skill architecture for financial evaluation with experience memory storing successful patterns and forbidden regions, used in a retrieve-generate-evaluate-distill loop.

  • Results

    FactorMiner constructs a diverse library of high-quality factors with competitive performance and low redundancy as the library scales across multiple datasets and markets.

  • Takeaways & Limitations

    The framework provides a lightweight approach to scalable discovery of interpretable formulaic alpha factors while retaining structural knowledge across mining sessions.

  • Takeaways & Limitations

    The evaluation targets predictive IC, ICIR, redundancy, and portfolio diagnostics; transaction-cost-aware deployment remains future work, and comparison baselines are not exhaustively tuned.

Abstract

from arXiv · show

Formulaic alpha factor mining is a critical yet challenging task in quantitative investment, characterized by a vast search space and the need for domain-informed, interpretable signals. However, finding novel signals becomes increasingly difficult as the library grows due to high redundancy. We propose FactorMiner, a lightweight and flexible self-evolving agent framework designed to navigate this complex landscape through continuous knowledge accumulation. FactorMiner combines a Modular Skill Architecture that encapsulates systematic financial evaluation into executable tools with a structured Experience Memory that distills historical mining trials into actionable insights (successful patterns and failure constraints). By instantiating the Ralph Loop paradigm -- retrieve, generate, evaluate, and distill -- FactorMiner iteratively uses memory priors to guide exploration, reducing redundant search while focusing on promising directions. Experiments on multiple datasets across different assets and Markets show that FactorMiner constructs a diverse library of high-quality factors with competitive performance, while maintaining low redundancy among factors as the library scales. Overall, FactorMiner provides a practical approach to scalable discovery of interpretable formulaic alpha factors under the "Correlation Red Sea" constraint.

1 Introduction

Formulaic alpha discovery must navigate a combinatorial expression space while preserving interpretability and accumulating knowledge across sessions. FactorMiner addresses these constraints with reusable mining skills, structured experience memory, and iterative library-aware exploration.

  • Motivation: Formulaic factor discovery faces combinatorial search complexity, weak knowledge accumulation, and requirements for transparent, auditable financial logic.Traditional automated methods may repeat trials because they do not retain reusable insights about successful and failed searches.
  • Research question: Formulaic factors combine market fields with 60+ operators while supporting human auditing and compositional generalization across market regimes.The central question is how to explore this program space while maintaining a global view of the factor library.
  • Approach: FactorMiner combines a compositional skill architecture with experience memory to address factor-mining challenges.The skill packages domain knowledge and validation procedures, while memory stores successful patterns and forbidden regions from historical sessions.
  • Approach: The Ralph Loop retrieves memory priors, generates candidates through the mining skill, evaluates them, and distills outcomes back into memory.This feedback cycle is intended to improve future exploration efficiency while preserving a global factor-library perspective.
  • System design: The system uses GPU acceleration, multiprocessing, and C-compiled numerical operations to make large-scale iterative evaluation feasible.The architecture separates agent reasoning from reusable skill execution and supports independent skill upgrades.
  • System design: FactorMiner incorporates library-admission mechanisms so mining decisions consider how candidates complement existing factors rather than optimizing factors in isolation.The resulting open library contains 110 A-share equity factors with explicit formulaic expressions.

2 Related Work

Prior work spans explicit formula libraries, evolutionary and reinforcement-learning search, and tool-augmented agents with memory. FactorMiner connects these directions by packaging factor mining as a reusable skill and retaining searchable experience across sessions.

  • Formulaic factor libraries: Released formulaic alpha libraries provide explicit expressions and can combine diversity with effectiveness, although their discovery processes are often undocumented.One cited library contains 101 formulas with 15.9% mutual correlations, while Alpha191 extends coverage to about 191 expressions.
  • Automated search: Genetic programming evolves executable factor programs through crossover and mutation but may converge slowly because syntactic operators provide limited semantic guidance.Its exploration can therefore be inefficient in the large expression space.
  • Automated search: Reinforcement learning treats metrics such as IC and ICIR as rewards, while neural and language-model systems generate or refine factors using redundancy-reduction strategies.These approaches can require additional training and repeated evaluation overhead, and persistent management of explored structure remains challenging.
  • Tool-augmented agents: Tool-augmented agents interleave reasoning, tool calls, observations, and iteration, motivating factor mining as an executable skill invoked on demand.This framing separates high-level agent reasoning from specialized domain procedures.
  • Agent memory: Memory-based agents store reflections, observations, plans, or lessons to condition future behavior and reduce catastrophic forgetting.FactorMiner applies this memory perspective to symbolic program synthesis by retaining structural search knowledge rather than only individual outputs.
  • Continual improvement: Meta-learning views prior episodes as sources of transferable search biases, while long-horizon memory systems address evolving tools, action spaces, and redundancy patterns.These perspectives motivate summarizing reusable patterns and avoiding repeatedly explored dead ends.

3 Methodology

FactorMiner formulates factor discovery as iterative construction of a diverse library under predictive-quality and correlation constraints. Its modular skill architecture and persistent experience memory support executable evaluation, memory-guided exploration, and continual refinement.

  • Problem Formulation: FactorMiner defines formulaic alpha factors as programs over typed operators that transform market data into cross-sectional predictive signals.The input is a tensor over assets, time, and features; operators compose into expression trees with admissible parameters.
  • Problem Formulation: Factor quality is measured with IC and ICIR, while factor redundancy is measured by time-average cross-sectional Spearman correlation.The correlation definition can be replaced by alternative dependence measures, including time-series, partial, or nonlinear dependence.
  • Problem Formulation: The objective is to maximize aggregate predictive quality for a factor library while enforcing a global correlation budget.As the library grows, the feasible region for new orthogonal factors shrinks, creating the paper’s “Correlation Red Sea” challenge.
  • Self-Evolving Search: Memory retrieval conditions candidate generation on the current library and prior rejection reasons, steering exploration toward higher-utility and lower-redundancy regions.The Ralph Loop repeatedly retrieves, generates, evaluates, and distills experience so the agent learns how to search rather than repeatedly exploring redundant regions.
  • Modular Skill Architecture: FactorMiner separates factor discovery into a reusable modular skill containing operators, validation, and execution interfaces for an agent.The skill uses 60+ operators and a check_ic → check_correlation → admit pipeline, with deterministic evaluation and configurable multi-market support.
  • Experience Memory: Experience memory stores mining-state information, successful patterns, forbidden directions, and strategic insights from historical sessions.Memory formation extracts successful and rejected patterns from trajectories, while evolution consolidates redundant entries and removes low-utility information.
  • Resulting Library: The released full A-share library contains 110 admitted factors whose correlation structure is mostly weak to moderate, with only a few localized high-dependence clusters.This figure illustrates the diversity targeted by the library-level admission process.

4 Experiments

Across A-share and cryptocurrency datasets, FactorMiner delivers competitive out-of-sample factor performance, controlled redundancy, cross-market robustness, and efficient large-scale evaluation. Its experience-guided search also increases candidate yield while filtering redundant signals.

  • Experimental Setup: FactorMiner is evaluated on A-share index constituents and 64 major Binance cryptocurrency assets using 10-minute bars.The A-share data covers CSI 500, CSI 1000, and CSI 300 universes, with over 25 million data points in aggregate.
  • Main Results: Under a strict frozen-Top-40 protocol, FactorMiner achieves IC/ICIR of 8.25%/0.77 on CSI500 and remains competitive across four markets.Top-40 factors are selected once on CSI500 during 2024 and evaluated out of sample during 2025.
  • Factor Quality and Diversity: Average pairwise absolute correlation is 0.30–0.31 on A-shares and 0.25 on Crypto, while correlation tails remain controlled rather than dominated by near-duplicate signals.The full released A-share library contains 110 admitted factors with average offdiagonal absolute correlation of 0.203.
  • Robustness Across Heterogeneous Markets: FactorMiner maintains competitive Crypto performance, reaching IC/ICIR of 3.82%/0.28 for the single-factor library and 9.48%/0.62 with IC-weighted combination.The evaluation spans a market with different trading hours, microstructure, and risk-factor structure from A-shares.
  • Robustness Across Heterogeneous Markets: Market heterogeneity motivates adaptive admission: lower-dispersion, higher-synchronicity Crypto benefits from lower cross-sectional thresholds and timing-oriented objectives.A-shares exhibit richer cross-sectional dispersion, making strict cross-sectional admission more meaningful there.
  • Portfolio Construction: For FactorMiner, learned selection adds limited gains over simple ensembles, with CSI500 ICIR of 1.29/1.31 for EW/ICW versus 1.21/1.29 for Lasso/XGBoost.This indicates that simple ensemble signals already capture most exploitable predictive power under the Train’24/Test’25 protocol.
  • Additional Robustness Checks: Against end-to-end predictors, FactorMiner reaches 8.25%/0.77 on CSI500 versus LightGBM’s 5.53%/0.51, while their low correlations suggest complementary information.The end-to-end outputs have average absolute correlation below 0.04 with the released 110-factor library and maximum single-factor correlation of 0.23.
  • Effect of Experience Memory: Experience memory raises high-quality candidate yield from 20.0% to 60.0% while increasing redundancy rejection from 43.8% to 55.2%.The Have Memory variant generates 96 high-quality candidates versus 32 for No Memory, supporting more directed and diverse exploration.

5 Discussion

FactorMiner combines interpretable factors, experience memory, and accelerated computation to support scalable high-frequency alpha discovery. The discussion identifies reusable success and failure patterns, while noting that orthogonal factors become harder to find as the library grows.

  • 110 interpretable high-frequency alpha factors and a standardized evaluation protocol support reproducible, hypothesis-driven market-microstructure analysis.
  • Experience memory enables continual learning by storing successful factor patterns and failure constraints across mining sessions.The system learns how to search more effectively, not only which individual factors perform well.
  • Higher-moment, trend-regression, and amount-efficiency interactions are identified as patterns associated with high-IC, low-correlation factors.
  • GPU computation delivers consistent order-of-magnitude gains over Python and optimized CPU backends for operator- and factor-level computation.Figure 4 reports computation time in milliseconds on a logarithmic scale, where lower values are better.
  • VWAP-deviation variants, standardized returns, and simple Delta reversals are identified as failure patterns that correlate with existing factors.

6 Conclusion

FactorMiner combines a modular mining skill with experience memory to discover interpretable high-frequency alpha factors. Its 110-factor library and standardized protocol provide a reproducible artifact for hypothesis testing, mechanistic inspection, and cross-market transfer studies.

  • FactorMiner combines a modular mining skill with experience memory in a lightweight self-evolving framework for interpretable high-frequency alpha discovery.
  • The resulting 110-factor library and standardized protocol support reproducible hypothesis testing, mechanistic inspection, and cross-market transfer studies.

7 Limitations and Ethical Considerations

The paper’s comparisons with end-to-end models are limited by incomplete baseline tuning, and deployment-oriented transaction-cost analysis remains outside the current pipeline. The authors also flag potential misuse and require compliance and risk controls.

  • End-to-end comparison baselines are not exhaustively tuned, limiting the strength of performance comparisons.
  • The pipeline evaluates predictive IC, ICIR, redundancy, and portfolio diagnostics, while transaction-cost-aware deployment remains future work.
  • Discovered factors could be misused in speculative or manipulative strategies, so deployment should follow compliance and risk-control requirements.

A Operator Library and Admission Criteria

The factor-mining skill represents symbolic factors as expression trees built from raw market features and more than 60 typed operators. Admission uses IC and correlation thresholds, with a replacement rule for sufficiently strong candidates, while accelerated backends support reproducibility.

  • Operator Library: Each symbolic factor is an expression tree over raw market features and a curated library of 60+ typed operators.Operators are organized into arithmetic, statistical, time-series, cross-sectional, smoothing, regression, and logical categories.
  • Operator Library: Table 7 organizes representative operators into arithmetic, statistical, time-series, cross-sectional, smoothing, regression, and logical categories.
  • Admission Criteria: A factor is admitted when it satisfies the specified IC and correlation criteria, using default thresholds τIC = 0.04 and θ = 0.5 for A-share mining.
  • Admission Criteria: A high-quality candidate can replace one correlated library factor when IC(α) ≥ 0.10, IC(α) ≥ 1.3 IC(g), and exactly one existing factor exceeds the correlation threshold.This rule allows library quality to improve while preserving diversity.
  • Reproducibility: Symbolic proposals use Gemini 3.0 Flash without weight updates, while factor evaluation uses GPU/C-accelerated operators and a 40-worker multiprocessing pool.Representative operator runtimes are reported in Figure 4, and Alpha101 is adapted to 10-minute bars.

B End-to-End Signal Correlations

The released 110-factor library has low correlation overlap with end-to-end model signals, while the models may still provide complementary information.

  • Average absolute correlations between PatchTST, Chronos-2, LightGBM signals and the 110-factor library are below 0.04.
  • The maximum single-factor correlation is at most 0.23 across these end-to-end models and the released library.
  • Pairwise correlations among the three end-to-end signals are also modest.
  • The end-to-end models do not dominate the formulaic library in predictive performance but may offer complementary information for hybrid systems.

C Experience Memory: Forbidden Directions

Experience memory records structural directions that repeatedly produce factors redundant with the existing library, especially within a dense VWAP-related cluster.

  • Experience memory distills negative knowledge about structural directions that repeatedly yield candidates highly correlated with the existing factor library.
  • The VWAP cluster, including factor 006 and variants, is the library’s densest correlation region.
  • Close-to-VWAP signals normalized by volatility or volume almost always correlate above 0.5 with factor 006 and its variants.
  • As the library exceeded 70 factors, even high-IC candidates increasingly collided with existing factors, suggesting that the easy signal space was largely exhausted.

D Full Factor Library (110 factors)

The factor library contains formulaic signals built from price, volume, return, VWAP, volatility, and distributional features using ranking, time-series, arithmetic, and conditional operators.

  • Several factors use cross-sectional ranking to transform price-relative, volume-relative, return, and volatility-derived quantities.
  • The library includes signals combining close, VWAP, high, low, open, volume, amount, returns, volatility, and distributional statistics.
  • Time-series operators such as Delta, TsRank, Std, Mean, Slope, and Rsquare appear across the factor formulas.
  • Conditional factors switch between trend, return, volatility, skewness, and VWAP-volume signals using threshold-based IfElse logic.
  • The formulas also include interactions that multiply or combine ranked signals from price position, returns, volume, skewness, kurtosis, and slope.
Loading 2602.14670v2…