Source-linked AI summary
Lineage-Aware Search Control for LLM-Based Evolutionary Factor Mining
Xintong Liu, Chenkai Zhang, Yanzhao Ma, Peng Yang
TL;DR
LLM-based evolutionary factor mining uses semantic variation, but current methods make limited use of lineage-level state to guide operators and organize search. LineageEvo conditions LLM variation on lineage experience and allocates search across lineages; it reports the best results on five metrics in both evaluated markets.
Problem
LLM-EA factor mining mainly organizes search at individual and population levels, leaving lineage-level state underused despite its potential to preserve local-search continuity.
Source: Introduction · pp. 1–2Method
LineageEvo uses lineage-scoped experience to form operator-specific priors for LLM variation and uses lineage-stratified parent selection to distribute search opportunities across lineages.
Source: Introduction · p. 2, Lineage-Conditioned LLM Variation · p. 4Results
LineageEvo achieves the best result on all five metrics in both CSI 500 and S&P 500.
Source: Main Results · p. 6Takeaways & Limitations
The results support lineage as an intermediate organizational unit for evolutionary factor mining that can accompany predictive performance with a more diverse factor pool.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Formulaic alpha factor mining searches for symbolic expressions that transform market data into predictive signals. Recent LLM-based evolutionary factor mining methods use large language models as semantic variation operators. Under the conventional LLM-EA paradigm, stochastic variation and fitness-based selection can leave the search weakly directed and concentrated around a narrow set of structurally similar factors. We argue that these limitations arise because current methods mainly organize search at the individual and population levels, leaving lineage-level control underused. A lineage connects successive factor variants through evolutionary descent. We propose LineageEvo, which uses lineage in two coupled forms of search control: (1) lineage-conditioned LLM variation, which distills Lineage-Scoped Search Experience into operator-specific priors and fuses them with global operator priors, providing persistent, lineage-relevant guidance for LLM generation; and (2) lineage-stratified search allocation, which balances factor quality against lineage concentration by distributing the search budget across lineages during evolutionary search. Together, these mechanisms make semantic variation more direction aware and prevent branches from dominating the evolution, thereby promoting the discovery of predictive and diverse factors. We evaluate LineageEvo on CSI 500 and S&P 500 in terms of factor predictiveness, portfolio performance, search efficiency, and lineage- and expression-level diversity. LineageEvo achieves the strongest predictive performance on both markets, while LSPS produces a more balanced lineage distribution and a structurally broader factor pool. The results support lineage as a useful intermediate unit for controlling LLM-based evolutionary factor search.
Introduction
LLM-based evolutionary factor mining offers semantic formula variation, but its operators can be weakly controllable and fitness selection can concentrate search on dominant branches. LineageEvo uses evolutionary descent to guide variation and allocate search across lineages, and reports stronger performance and broader diversity across two markets.
Introduction
- LLM-based evolutionary methods combine semantic variation with fitness selection, but their operators can be weakly controllable and dominant branches can monopolize search.These search-control challenges can reduce diversity and risk premature convergence under limited budgets.Source: Introduction · p. 1
- Current methods mainly organize search at individual and population levels, while making limited use of lineage-level state.A lineage is a descent-defined factor family that preserves local-search continuity across successive variation events.Source: Introduction · p. 1
- The study identifies lineage as an underused shared scope for operator priors and search allocation in LLM-EA factor mining.Experiments cover CSI 500 and S&P 500 and report that LineageEvo reaches a strong factor set rapidly.Source: Introduction · p. 2
- LineageEvo represents evolutionary descent in a lineage DAG and uses lineage-scoped experience to condition LLM mutation and crossover with operator-specific priors.It fuses these lineage priors with global operator priors; LSPS allocates search opportunities across lineages while a separate elite archive preserves high-quality discoveries.Source: Introduction · p. 2
- LineageEvo achieves the best result on all five metrics in both CSI 500 and S&P 500.The reported metrics cover predictiveness and portfolio performance; LSPS also increases lineage entropy from 0.405 to 2.154 and average AST distance from 0.509 to 0.747.Source: Introduction · p. 2
Related Work
Prior work expands automated factor search through symbolic methods, LLM-based generation, and evolutionary optimization. LineageEvo builds on lineage research by using lineages both to accumulate operator priors and to allocate search opportunities.
Traditional Automated Factor Mining
- Traditional automated factor mining includes GP, reinforcement learning, and structure-aware exploration, but largely relies on predefined grammars, handcrafted operators, or scalar rewards.These approaches make it difficult to incorporate domain semantics or adaptively reuse experience beyond predefined operators.Source: Traditional Automated Factor Mining · p. 2
LLM-Based Factor Mining
- LLM-based factor-mining methods use semantic generation, refinement, interactive agents, tree search, and graph-based retrieval.AlphaBench suggests existing methods may still under-exploit LLM exploration, motivating iterative population-based search.Source: LLM-Based Factor Mining · p. 2
LLM-EA Factor Mining
- LLM-EA methods add iterative evolutionary variation and selection to factor search, including factor evolution, tree-structured thought, and trajectory-level experience reuse.They still face weakly controllable LLM variation and the risk that fitness selection lets dominant branches monopolize search.Source: LLM-EA Factor Mining · p. 2
Genealogy and Lineage in Evolutionary Algorithms
- Earlier evolutionary-computation work used genealogy and lineage for diversity preservation, inheritance analysis, and selection; LineageEvo uses each lineage to scope priors and allocate search.This applies lineage as a shared control unit in LLM-EA factor mining.Source: Genealogy and Lineage in Evolutionary… · p. 2
Method
LineageEvo uses lineage as an intermediate search unit that scopes accumulated operator experience and allocates future search opportunities. Its loop combines lineage-stratified parent selection, lineage-conditioned LLM variation, and updates to the evolutionary state.
Method
- LineageEvo associates accumulated search experience and future opportunities with each lineage, linking individual-level LLM variation to population-level selection.The framework represents evolutionary descent in a lineage DAG and separates semantic variation from deterministic validation, evaluation, and state transitions.Source: Method · pp. 2–3
Problem Setup and Lineage Representation
- The target is a diverse collection of factors with strong predictive performance and temporal stability, rather than a single best expression.Lineages follow primary inheritance of the expression backbone; crossover donor ancestry is retained separately through secondary edges.Source: Problem Setup and Lineage Representation · pp. 3–4
Lineage-Stratified Search Allocation
- LSPS selects a lineage before a parent, combining lineage quality with greater weight for less represented lineages while retaining quality-based parent sampling.Its quality score penalizes disagreement between training and validation strength and directional reversals.Source: Lineage-Stratified Search Allocation · p. 4
Lineage-Conditioned LLM Variation
- Lineage-conditioned variation uses separate mutation and crossover priors, updates guidance from evolutionary events, and fuses lineage-specific and global experience into LLM context.The fusion shifts emphasis toward a lineage’s prior during improvement and toward the global prior under stagnation or degradation; the context is soft guidance, not a hard constraint.Source: Lineage-Conditioned LLM Variation · pp. 4–5
Feedback and Search-State Update
- Each valid offspring is evaluated and recorded as an event, after which the lineage DAG, active population, elite archive, and eligible priors are updated.Invalid candidates do not enter the DAG or consume the valid-evaluation budget.Source: Feedback and Search-State Update · p. 5
Experiments
Across CSI 500 and S&P 500, the experiments evaluate factor predictiveness, portfolio performance, diversity, and search efficiency. LineageEvo leads the reported main results, while ablations and search curves examine its priors, LSPS, and discovery speed.
Experiments
- Experiments test whether lineage-aware search improves predictive and stable factors, prior guidance, diversity, concentration, and search efficiency.IC and ICIR measure predictiveness and temporal stability; portfolio measures include annualized return, information ratio, and maximum drawdown.Source: Experiments · p. 5, Experimental Setup · p. 5
Experimental Setup
- The evaluation covers CSI 500 and S&P 500 with a shared temporal split and compares traditional, LLM-based, and LLM-EA baselines.The training period is 2015-01-01 to 2020-12-31, validation is 2021-01-01 to 2022-04-30, and testing is 2022-05-01 to 2026-04-30.Source: Experimental Setup · p. 5
Main Results
- LineageEvo leads all five metrics in both markets and finishes with the highest cumulative return among compared methods on CSI 500.The main-results comparison reports factor and portfolio metrics; the cumulative-return curve covers the CSI 500 test period.Source: Table 1 · p. 6, Main Results · p. 6, Figure 2 · p. 6
Ablation on Search Priors
- Removing all priors lowers CSI 500 IC from 0.0512 to 0.0384 and ICIR from 0.2740 to 0.1994; lineage-scoped experience provides the main useful guidance.The ablations also identify important roles for both mutation and crossover priors.Source: Ablation on Search Priors · pp. 6–7, Table 2 · p. 7
Factor Diversity and Search Concentration
- LSPS raises IC from 0.0381 to 0.0512 and ICIR from 0.2415 to 0.2740 while increasing all three diversity measures.The authors describe LSPS as balancing lineage quality against concentration, preserving promising branches without letting them monopolize search.Source: Table 3 · p. 7, Factor Diversity and Search… · p. 7
Search Efficiency
- LineageEvo maintains a higher top-5 mean-so-far IC through most of the search and reaches a strong plateau within about 75 valid evaluations.RD-Agent improves quickly but saturates at a relatively low level; AlphaPROBE improves more gradually and requires more evaluations to reach a stronger factor set than RD-Agent.Source: Figure 3 · p. 7, Search Efficiency · p. 7
Conclusion
LineageEvo uses lineage as an intermediate unit for evolutionary factor mining, combining lineage-conditioned priors with lineage-stratified parent selection. The framework discovers predictive and stable factors, achieves strong investment returns, and preserves a more diverse factor pool.
Conclusion
- LineageEvo combines lineage-conditioned priors for LLM variation with Lineage-Stratified Parent Selection to distribute search opportunities across lineages.The paper identifies lineage as an intermediate organizational unit for evolutionary factor mining.Source: Conclusion · p. 7
- The framework discovers highly predictive and stable factors, achieves strong investment returns, and preserves a more diverse factor pool.Source: Conclusion · p. 7
- Future work may compare lineage with clusters, niches, and retrieval neighborhoods to study their advantages in experience reuse and search control.Source: Conclusion · p. 7
A Case Study
The case study traces how successful mutations on one lineage update guidance that is reused on a sibling branch. The example combines lineage-specific experience with patterns observed across lineages.
A Case Study
- At generation 30, replacing TsSkew with TsCov between close and high returns increased validation |IC| from 0.0209 to 0.0369.The event triggered both lineage and global prior rewriting.Source: A Case Study · p. 8, Figure 4 · p. 9
- On a sibling branch, the updated prior guided the same TsSkew-to-TsCov replacement while preserving its denominator, strengthening the pattern from medium to high confidence.The figure marks the prior transfer with a dashed connector.Source: A Case Study · p. 8, Figure 4 · p. 9
- Lineage guidance captures factor-family-specific successful and failed transformations, while global guidance contributes patterns observed across lineages.Their relative emphasis is set by the fusion policy before guidance enters the candidate-generation prompt.Source: A Case Study · pp. 8–9
B Metric Definitions
The evaluation defines factor-level predictiveness and stability, portfolio performance, search efficiency, and lineage- and expression-level diversity measures. It also specifies validation-based factor orientation and test-statistic separation.
B Metric Definitions
- Portfolio metrics are annualized return (AR), information ratio (IR), and maximum drawdown (MDD), with signed MDD values closer to zero treated as better.The reported portfolio metrics follow Qlib evaluation practice.Source: B Metric Definitions · pp. 9–10
- IC is the temporal mean of daily cross-sectional Spearman correlation between factor values and future returns, while ICIR measures the temporal stability of daily IC.Under the stated implementation, IC is equivalent to RankIC.Source: B Metric Definitions · pp. 9–10
- Factor direction is fixed using validation information before test evaluation, and test statistics are not used for orientation, selection, or prior updating.Source: B Metric Definitions · p. 10
- Search efficiency is summarized by validation IC over valid evaluations; invalid and duplicate attempts do not advance the evaluation count.Source: B Metric Definitions · p. 10
- Lineage entropy measures population allocation across lineages, with higher entropy indicating a more balanced distribution.Source: B Metric Definitions · p. 10
- Expression diversity uses normalized AST tree-edit distance, while effective structure counts treat identical normalized AST signatures as the same group.The structural groups are exact normalized AST structures, not distance-based clusters or partial motifs.Source: B Metric Definitions · p. 10
C Baseline Descriptions
The comparison includes traditional factor-mining, LLM-based, and LLM-EA baselines, alongside a hand-crafted reference. The described methods span evolutionary programming, reinforcement learning, structure-aware exploration, and LLM-driven search.
C Baseline Descriptions
- The baselines represent traditional factor mining, LLM-based factor mining, and LLM-EA factor mining.Source: C Baseline Descriptions · p. 10
- Alpha158 is a standard hand-crafted Qlib factor set used as a non-search reference, while GP is a classical non-LLM evolutionary baseline.GP uses symbolic mutation, crossover, and fitness selection.Source: C Baseline Descriptions · p. 10
- AlphaGen generates synergistic formulaic alpha collections through reinforcement learning, and AlphaSAGE uses GFlowNet-based structure-aware exploration.Source: C Baseline Descriptions · p. 10
- RD-Agent is an LLM-based multi-agent framework for data-centric factor and model optimization, while AlphaPROBE combines retrieval with on-graph biased evolution.Source: C Baseline Descriptions · p. 10
- EoH iteratively generates, evaluates, and evolves heuristic programs with LLMs, while ReEvo uses reflective evolution for LLM-based heuristic search.Source: C Baseline Descriptions · p. 10
- Table 4 summarizes the main implementation and evaluation parameters.Source: Table 4 · p. 11
D Experimental Protocol and Implementation Details
LineageEvo is evaluated on CSI500 and S&P 500 using daily data, temporal train/validation/test splits, and a shared Qlib pipeline. Its protocol combines lineage-aware allocation and variation with validation-based factor selection and a common portfolio backtest.
Markets, target, and temporal splits
- 2015–2020 training, 2021–April 2022 validation, and May 2022–April 2026 testing are used for both markets; test data are accessed only after selection is fixed.The prediction target is the next-period return, implemented as Ref($close,-2)/Ref($close,-1)-1.Source: D Experimental Protocol and… · p. 11
Lineage-stratified sampling
- Lineage allocation uses lineage quality and size, and the population updater caps any one lineage at 40% of active factors.Mutation selects a lineage and parent; crossover selects an anchor lineage and parent, preferentially draws a partner from another lineage, and assigns the child to the primary parent's lineage.Source: D Experimental Protocol and… · p. 11
Operator priors and rewriting
- Lineage-specific mutation and crossover priors are fused with global operator priors as soft guidance for candidate generation.Prior rewriting is event-triggered after candidate evaluation, updates a complete structured state when an improvement or degradation threshold is met, and otherwise leaves the prior unchanged.Source: D Experimental Protocol and… · p. 12
Final factor selection and orientation
- Final factors are selected primarily from the elite archive using q(f), deduplicated after normalization, and oriented by validation metrics without test statistics.Top-K selection takes the K highest-scoring distinct factors; for K > 1, oriented values are normalized cross-sectionally each day and equally weighted.Source: D Experimental Protocol and… · p. 12
Backtesting protocol
- Portfolio evaluation uses Qlib's TopkDropoutStrategy with topk = 50, n_drop = 5, risk_degree = 0.95, daily execution, and shared transaction costs within each market.AR, IR, and MDD are computed from cost-adjusted returns.Source: D Experimental Protocol and… · p. 12
Ablation configurations
- Prior ablations remove selected lineage or global operator guidance, while the LSPS ablation replaces lineage-first allocation with score-weighted sampling over the full population.The ablations preserve the search budget, LLM backend, validation protocol, and non-ablated components.Source: D Experimental Protocol and… · p. 12
E Prompt Templates
The prompt templates standardize candidate generation while supplying operator-specific instructions and search context. They cover seed generation, mutation, crossover, and structured prior rewriting.
Shared prompt design
- All candidate-generation prompts share the factor DSL, look-ahead prevention rules, expression-length constraints, and JSON-format requirements.Each LLM request contains one system message and one user message, and returns exactly one candidate expression.Source: E Prompt Templates · p. 12
Seed-generation prompt
- Seed generation asks for a diverse, economically plausible factor based on a distinct financial hypothesis and the permitted DSL.The seed prompt requests a factor name, hypothesis, expression, expected signal type, rationale, and risk notes.Source: E Prompt Templates · pp. 12–13
Mutation prompt
- Mutation conditions generation on one parent, lineage and global mutation priors, and program-computed search-control variables.The prompt treats priors as soft guidance and directs the LLM to perform one coherent targeted mutation.Source: E Prompt Templates · p. 13
Crossover prompt
- Crossover uses the primary parent as the expression backbone and imports a local subtree or structural motif from the secondary parent.The prompt includes experience from both parents, with fusion centered on the primary lineage, and disallows copying the complete secondary expression.Source: E Prompt Templates · pp. 13–14
Prior-rewriting prompt
- Prior rewriting is triggered by informative evolutionary evidence and returns a complete structured prior that is checked against a deterministic schema.Lineage priors retain operator-specific successful or harmful patterns, while global priors retain patterns considered transferable across lineages.Source: E Prompt Templates · p. 14