Source-linked AI summary
FaVOR: LLM-Based Agentic Framework for Factor Mining via Empirical Validation
Hyeonjin Kim, Minseok Kim, Seunghyeon Jung, Sujin Pyo, Huisu Jang, Woojin Lee
TL;DR
Return-oriented LLM factor mining often lacks evidence that generated formulas preserve their intended economic mechanisms, limiting confidence in their robustness across regime shifts. FaVOR introduces a three-stage process that decomposes hypotheses, validates factors against observable conditions, and integrates validated factors. In 2025 out-of-sample tests on the CSI 500 and S&P 500, it delivers consistent performance across both markets.
Problem
LLM-based factor mining provides limited evidence that a factor’s mathematical form reflects its intended economic mechanism, leaving genuine signals difficult to distinguish from opaque correlations.
Method
FaVOR decomposes hypotheses into observable conditions, validates candidate factors against those conditions, and integrates factors that pass the validation stages.
Results
FaVOR delivers consistent 2025 out-of-sample performance across the CSI 500 and S&P 500, including cumulative excess returns of 0.2225 and 0.1123, respectively.
Takeaways & Limitations
FaVOR produces structurally interpretable factors that combine empirical performance with economic rationale.
Takeaways & Limitations
Directional Selectivity is not designed for reversal-type signals.
Abstract
from arXiv · showhide
Traditional finance relies on experts to hand-craft factors through a principled process grounded in economic rationale. Recent LLM-based multi-agent systems have automated this process, scaling factor mining far beyond manual effort. However, these automated approaches optimize directly for returns and rarely check whether a generated factor still expresses the economic hypothesis that motivated it. We identify this inconsistency between mathematical form and economic meaning as a structural failure mode of return-oriented automation. The resulting factors blur the line between real signals and spurious correlations and break down across regime shifts. We propose FaVOR (Factor Validation through Observable Reasoning), an agentic framework that restructures factor mining around hypothesis-level evidence rather than return outcomes. In place of the standard hypothesis-to-formula leap, FaVOR enforces a three-stage consistency loop tying mathematical form to economic rationale throughout. (1) Decomposition splits a broad economic hypothesis into independent observable conditions. (2) Validation checks whether each factor reflects its intended condition. (3) Integration merges them into a composite whose structure remains interpretable. On the CSI 500 and S&P 500 in 2025, FaVOR outperforms existing baselines while remaining effective across regimes. FaVOR shows that hypothesis-grounded factor discovery produces signals that are interpretable by construction, regime-robust, and economically faithful. The code is available at https://github.com/damilab/FaVOR.
1 Introduction
LLM-based factor mining scales candidate generation but often lacks evidence that formulas reflect their intended economic mechanisms. FaVOR addresses this gap through decomposition, factor-level validation, and integration, with consistent 2025 out-of-sample results across two equity markets.
- Motivation: LLM-based approaches can produce formulas that fit opaque in-sample correlations yet break down when market regimes shift.The problem is limited evidence that mathematical form reflects the intended economic mechanism.
- Contribution: FaVOR combines scalable factor mining with pre-backtest validation using distributional evidence.This contrasts with traditional manual construction and return-oriented automated mining.
- Contribution: FaVOR decomposes economic hypotheses into observable conditions, validates candidate factors against those conditions, and integrates validated factors.The framework admits factors only after they pass all three stages.
- Results: 0.2225 cumulative excess return and 1.5295 IR were achieved on the CSI 500 after transaction costs in the 2025 out-of-sample window.The corresponding S&P 500 results were 0.1123 cumulative excess return and 1.1315 IR.
2 Related Work
Factor research traditionally links formulas to explicit economic mechanisms through empirical validation, whereas newer LLM systems automate factor generation and optimization. FaVOR targets the remaining gap between automated generation and evidence that factors reflect underlying market states.
- Traditional Foundations: Traditional factor research treats factors as measurements of economic mechanisms rather than formulas optimized only for returns.Researchers use empirical validation to test alignment with explicit hypotheses.
- FaVOR: FaVOR decomposes hypotheses, validates factors against target conditions, and integrates the subset that best reconstructs the hypothesis.The framework reports stable and interpretable performance, with ablations identifying Validation and Integration as essential.
- Automated Factor Mining: LLM-based systems automate candidate generation by translating investment ideas into formulas and coordinating specialized agents.Examples include FAMA, Alpha-GPT, and RD-Agent-Quant.
- Research Gap: Natural-language explanations and realized performance checks do not establish that a factor reflects an underlying market state.This leaves a gap between automated factor generation and traditional empirical validation.
3 Methods
FaVOR converts hypotheses into observable conditions, generates candidate formulas, validates their economic consistency with distributional evidence, and integrates validated combinations into executable signals. Its integration stage tests whether selectivity strengthens realized signal quality before threshold optimization and backtesting.
- Stage 1: Decomposition: FaVOR generates candidate factors by decomposing a high-level hypothesis into distinct observable market conditions.The hypothesis, observation, and factor agents perform these transformations.
- Stage 2: Validation: Factor-level validation uses factor-conditioned market-state statistics rather than future returns to test whether formulas measure their intended conditions.Stocks are sorted into quintile bins and described using OHLCV-based variables and descriptive statistics.
- Stage 2: Validation: Validation decisions combine systematic central-tendency shifts, tail-spread variation, agreement across statistics, and semantic agreement with the observation.The final pass/fail decision is deterministic once the observation-to-variable mapping is supplied.
- Stage 3: Integration: Validated factor sets are combined through a Cartesian product, selecting one factor for each observable condition.The resulting combinations move the analysis from individual factors to joint realizations of the original hypothesis.
- Stage 3: Integration: A joint trading signal triggers only when every constituent factor crosses its polarity-specific in-sample quantile threshold simultaneously.Increasing σ makes the signal more selective and retains stronger realizations of the hypothesized market state.
- Stage 3: Integration: Stricter selectivity produces higher average realized returns and win rates for supported combinations.Combinations are retained only when ticker-level support exceeds a fixed pass-rate threshold.
- Stage 3: Integration: Threshold optimization uses a held-out validation set and Calmar ratio before fixed-threshold testing on the held-out test period.The training-set selectivity sweep performs structural screening, while factor-specific execution thresholds are optimized separately.
4 Experiments
Experiments evaluate FaVOR on two equity universes using excess-return backtests, factor-level validation cases, ablations, and backbone sensitivity analyses. FaVOR achieves the strongest reported performance across both markets while validation and integration materially support its trading results.
- Main Results: FaVOR achieves the best result across all four metrics on the CSI 500: AR 0.2067, IR 1.5295, MDD −0.0853, and CR 0.2225.These results simultaneously improve return generation and reduce downside risk relative to every compared method.
- Main Results: FaVOR again ranks best across all four metrics on the S&P 500, reaching AR 0.1062, IR 1.1315, MDD −0.0443, and CR 0.1123.The headline numbers are reported as reproducible across 10 independent runs.
- Main Results: Cumulative-return trajectories show FaVOR preserving early CSI 500 gains and compounding from mid-year on the S&P 500 while most baselines remain weak or negative.The comparison uses cumulative excess returns for both markets.
- Factor-level Validation: Factor-level validation separates passing and failing candidates through distributional behavior, with consistent PASS/FAIL contrasts across intraday recovery, post sell-off, and early stabilization observations.Passing factors exhibit systematic shifts and stronger tails consistent with their intended market condition; failing factors move oppositely.
- Impact of Validation and Integration: Removing Stage 2 more than doubles MDD, removing Stage 3 causes the largest return deterioration, and removing both stages makes every metric negative.The integration ablation attributes the largest return loss to validated factors firing individually rather than through jointly checked combinations.
- Sensitivity to LLM Backbone: FaVOR retains positive AR and IR across all evaluated backbones and markets, although performance dispersion shows that backbone choice still affects factor strength and robustness.On the CSI 500, open-weight Qwen3-235B achieves IR 1.0598; on the S&P 500, GPT-4o ranks first across all four metrics.
5 Conclusion
FaVOR grounds factor mining in structural validity by decomposing hypotheses, validating candidate factors, and integrating those that reconstruct the hypothesis. On the CSI 500 and S&P 500, it delivers stable, interpretable performance, with Validation and Integration each confirmed as essential.
- FaVOR decomposes hypotheses into observable market conditions, validates factors against those conditions, and integrates the subset that best reconstructs the hypothesis.
- FaVOR delivers stable and interpretable performance against competitive baselines on the CSI 500 and S&P 500.
- Ablations confirm that the Validation and Integration stages are each essential.
Limitations
The paper’s scope excludes reversal-type signals whose direction flips under stricter thresholds, and its daily-bar cost model assumes negligible market impact and slippage at studied trade sizes and holding periods.
- Reversal-type signals whose direction flips under stricter thresholds are outside FaVOR’s selection scope.The criterion targets combinations whose predicted direction remains consistent as thresholds become stricter.
- The daily-bar backtests treat market impact and slippage as typically negligible for the studied trade sizes and holding periods.The reported average holding period is 9.64 days, while higher-frequency applications may require a high-fidelity execution simulator.
Ethical Considerations
FaVOR uses daily OHLCV data and constrained, deterministic factor-construction and validation procedures. Candidate factors are retained only when multiple empirical consistency checks jointly support their intended observable conditions.
- The study uses aggregated daily OHLCV data without personally identifiable or proprietary information in prompts.LLM outputs are verified against historical return statistics rather than accepted at face value.
- Observation functions use a predefined library of deterministic operators, and formula construction is restricted to those operators applied to raw OHLCV inputs.The library includes normalization, time-series transformations, smoothing, mathematical primitives, conditional logic, regression utilities, and technical indicators.
- Table 5 documents the operator library used for observation-function construction.
- Validation tests central-tendency shifts, tail behavior, statistical consistency, and semantic consistency against each observable market condition.The checks use complementary empirical statistics and compare observed patterns with the textual meaning of the condition.
- A candidate factor is retained only when all four validation criteria jointly hold.The validation agent applies the checks to the candidate factor’s statistical profile.
C Evaluation Metrics
The evaluation uses portfolio-level metrics to assess profitability, risk-adjusted excess performance, and downside risk over time, alongside fixed and shared experimental configurations.
- Evaluation Metrics: The annualized excess return captures geometric annualized strategy performance over the benchmark.
- Evaluation Metrics: The Information Ratio measures annualized excess performance per unit of benchmark-tracking risk.Higher IR indicates more efficient excess-return generation per unit of tracking risk.
- Evaluation Metrics: Maximum drawdown measures the largest peak-to-trough decline in the portfolio’s own cumulative wealth.Values closer to zero indicate better downside risk control and capital preservation.
- Evaluation Metrics: Cumulative Return measures total excess return over the full investment horizon relative to the benchmark, without annualization.The active return is defined as portfolio return minus benchmark return.
- Experimental Configuration: FaVOR uses deterministic design parameters, while data splits, transaction costs, execution rules, stop-loss policy, and baseline workflows are shared across methods.
- Agent Pipeline: The pipeline uses specialized language-model agents for hypothesis construction, observation decomposition, formula generation, and construct validation.
G Strategy-Level Statistics
Strategy-level evidence includes trade-distribution analysis and regime-performance visualization for FaVOR’s out-of-sample portfolios.
- Trade-Level Statistics: Trade-level statistics verify that FaVOR’s CSI 500 cumulative excess return is broadly distributed rather than driven by a few extreme trades.
- Regime Visualization: Figure 6 plots 2025 CSI 500 and S&P 500 market regimes against FaVOR’s cumulative return, using the right axis with a base of 1.0.
H Market-Regime Context for OOS 2025
The 2025 out-of-sample tests span distinct market paths, while additional analyses examine whether semantic validation improves strategy-level outcomes and rejects economically mismatched formulas.
- Market-Regime Context: Regime labels use trailing 60-day benchmark returns and are applied only for post-hoc interpretation, not factor selection or trading.
- Market-Regime Context: CSI 500 experienced consolidation followed by a second-half rally, whereas the S&P 500 underwent a drawdown followed by recovery.
- Validation Ablation: Removing Stage 2 factor-level validation deteriorates excess IR in both markets while other settings remain fixed.
- Validation Ablation: PASS formulas show no uniform standalone RankIC advantage, consistent with Stage 2 serving as a semantic and distributional consistency filter.
- Formula Consistency: The candidate -TS_SUM(DELTA(low,1),5) has positive RankIC across both markets and examined horizons, yet represents continuing new lows rather than weakening downside pressure.
- Formula Consistency: Stage 2 rejects formulas whose induced OHLCV distributions contradict the market condition stated in the hypothesis.
J Auditability and Reliability of Semantic Validation
FaVOR makes semantic validation auditable through explicit statistical evidence and rule-constrained judgment, then evaluates reliability against coded checks and blinded experts.
- Auditability: Validation combines deterministic factor quantile bins and OHLCV statistics with a rule-constrained semantic judgment.
- Reliability Study: The blinded reliability set contains 100 cases, balanced between 50 framework PASS and 50 framework FAIL decisions from 44 observation groups.
- Reliability Study: The reliability analyses compare validator interpretations with an independently coded checklist and blinded expert assessment rather than portfolio returns.
- Expert Assessment: Experts judged 39 cases as agreement, 49 as partial agreement, and 12 as disagreement.
- Expert Assessment: Partial agreement generally preserved observable-condition and directional alignment, while leaving the intermediate market mechanism insufficiently established.
- Scope Boundary: The validator tests consistency between formulas and factor-conditioned OHLCV evidence, but does not establish a causal or fundamental economic mechanism.
K Longitudinal Evaluation Across Test Years
The evaluation fixes combinations selected on historical validation data and tests them across multiple years and rolling windows, including varied market environments. Overlapping windows are treated as sensitivity checks rather than independent trials.
- Evaluation design: Fixed formula combinations and quantile levels are selected using 2015–2019 training and 2020 validation data, then held constant in subsequent tests.Only per-ticker cutoffs are estimated from the immediately preceding validation window and frozen during each test period.
- Evaluation design: Five-year testing spans the 2022 market decline and subsequent market environments, with annualized excess return and excess IR reported for each year.
- Evaluation design: Table 17 reports yearly performance for combinations selected before all reported test years, using preceding validation windows to set and freeze test cutoffs.
- Evaluation design: 49 rolling 12-month test windows assess sensitivity to the monthly start date, although adjacent windows overlap and are not independent trials.Each window follows a trailing 24-month validation period.
L Hypothesis Provenance and Failure Analysis
FaVOR generates many hypotheses but admits only a selective subset through its validation pipeline, revealing a scope tied to OHLCV-identifiable conditions. Benchmark comparisons use matched data and execution settings, but search capacities differ and interpretability remains empirical rather than causal.
- Hypothesis provenance: 520 unique hypotheses emerged from ten seed concepts across 696 outer-loop iterations, while Table 19 tracks progression toward positive out-of-sample combinations.
- Failure analysis: Only 49 of 696 hypothesis iterations cleared the Stage 3 gate, with failures concentrated in mechanisms that daily OHLCV data cannot cleanly identify.
- Failure analysis: FaVOR targets hypothesis–formula consistency for OHLCV-observable conditions rather than recovery of every economically plausible trading mechanism.
- Selectivity scope: Directional Selectivity rejected 81 of 212 Stage 3 combinations for unstable realized direction, including 47 that still had positive after-cost cumulative returns.The filter particularly excludes rebound and panic-selling patterns.
- Baseline context: Benchmark results use the same OHLCV inputs, splits, transaction costs, and backtest environment, but the methods have different inductive priors and search capacities.FaVOR uses language-model hypothesis priors, whereas ML/DL and RL baselines search predefined or OHLCV primitive spaces.
- Interpretability boundary: FaVOR’s interpretability claim concerns empirical consistency between formulas and motivating OHLCV conditions, not proof of a causal or fundamental economic mechanism.
- Implementation context: The implementation uses Qlib for simulation and transaction-cost evaluation, Optuna for Stage 4 threshold search, and multiple commercial or open-weight language-model backbones.Experiments also rely on package-based CSI 500 and S&P 500 price and volume data under stated licensing and provider restrictions.