Source-linked AI summary
Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research
Houde Qian, Zheng Jiang, Wenyi Li, Chenxi Xu, Xinjie Lin, Yuzhao Peng, Ruoyu Han, Youjie Zheng, Kaisen Yang, Qinhuai Na, You Fu, Calvin Xiao
TL;DR
Existing benchmarks assess parts of quantitative research, leaving the full strategy-building loop on real market data unevaluated. Quant-Arena benchmarks that loop with bounded research-period feedback and sealed assessment; none of nine frontier models beats the equal-weight market in 2024.
Problem
Existing benchmarks assess code, factor generation, or sequential trading decisions, while evidence on end-to-end feature construction and frozen-strategy assessment on real market data is limited.
Source: §1 · p. 2Method
Quant-Arena has models construct features and stock scores from real A-share data, revise candidates with bounded 2022–2023 feedback, and freeze a strategy for sealed 2024 assessment under common trading rules.
Source: §1 · p. 2Results
Across nine frontier models, none beats the equal-weight market in 2024, and the strongest model gains mainly by staying out rather than selecting better stocks.
Source: §1 · p. 2Takeaways & Limitations
A predictive correlation alone is insufficient to judge an investable strategy because top-ranked holdings can earn less than middle-ranked names.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Quantitative research is an iterative loop: a researcher proposes a market hypothesis, implements it as an executable signal, tests it on historical data, and decides what to keep. Existing benchmarks score formulas, code, or individual trading decisions, leaving open whether large language models (LLMs) can complete this loop and produce strategies that work beyond the research period. We introduce Quant-Arena, a benchmark and public leaderboard for end-to-end quantitative research on real A-share market data. A model constructs features and stock-scoring code, receives feedback only on 2022--2023, and freezes one strategy for assessment on the sealed year 2024. A cash-account evaluator applies trading constraints and costs, including market impact estimated from a simulated order book. Scores are dominated by realized excess return and complemented by signal quality and the diversity of features used by the strategy. We evaluate nine frontier models, among which GPT-6 Astra scores highest; yet no model beats the equal-weight market in 2024. Quant-Arena is open to submissions from any model or agent harness.
1 Introduction
Quant-Arena evaluates the full quantitative-research loop on real A-share data, from feature construction through frozen-strategy assessment. In 2024, no evaluated model beats the equal-weight market, and the strongest model gains mainly by staying out of the market rather than selecting better stocks.
1 Introduction
- Unlike benchmarks covering separate workflow parts, Quant-Arena evaluates feature construction, signal combination, and strategy selection jointly, from raw order and trade records through bounded feedback to frozen out-of-sample assessment.The authors describe it as the first such benchmark on real market data under common trading rules.Source: §1 · p. 2
- Models construct features and stock scores from raw market records, revise candidates using 2022–2023 feedback, then freeze one strategy for assessment on sealed 2024 data.Figure 2 depicts this progression under common portfolio and execution rules.Source: §1 · p. 2, Figure 2 · p. 3
- Shared portfolio and cash-account rules apply lot-size constraints and order-dependent costs, including market impact estimated from a simulated order book.Evaluation combines profitability measures with risk, signal statistics, and feature diversity.Source: §1 · p. 2
- A bounded research budget and unseen assessment address the risk that repeated backtests select candidates fitted to historical noise.The platform records candidate evaluations and submissions to examine how models use feedback and select strategies.Source: §1 · p. 2
- No evaluated model beats the equal-weight market in 2024; the strongest model gains mainly by staying out of the market rather than selecting better stocks.The study evaluates nine frontier models.Source: §1 · p. 2
- The public arena accepts submissions from any model or agent harness under the same protocol and sealed assessment.Source: §1 · p. 2
2 The Quant-Arena Benchmark
Quant-Arena evaluates out-of-sample profitability of stock-selection strategies developed from historical market data. Models construct and freeze strategies under a bounded research process, then a shared evaluator assesses them on sealed data using financial and signal metrics.
2.1 Task Formulation
- A strategy maps authorized historical observations H_t to daily stock scores s_t, which a common financial evaluator converts into portfolio positions and returns.The final submission is the frozen strategy program f.Source: §2.1 · p. 3
2.2 Market Data and Benchmark Construction
- The benchmark uses a fixed market-data release, stock universe, and trading calendar, with date-, instrument-, and field-restricted access under the same rules in research and assessment.Models construct features from raw inputs without a predefined factor library; missing observations and delisted stocks remain in the data.Source: §2.2 · p. 3
2.3 Research and Assessment Protocol
- Models may test at most five candidates on the research period, revise code using aggregate feedback, then freeze one strategy for a single sealed out-of-sample assessment.Assessment observations and feedback are not returned for further development.Source: §2.3 · p. 4
2.4 Financial Evaluation and Metrics
- The evaluator turns stock rankings into an equal-weight 75-stock portfolio and compounds returns under cash-account, lot-size, trading-cost, and simulated-order-book market-impact constraints.Idle cash earns zero return; daily net asset value includes cash and outstanding positions.Source: §2.4 · p. 4
- The benchmark reports out-of-sample return and excess return alongside risk and signal metrics; its quality score weights realized excess return most heavily, while feature diversity also contributes.The reported metrics include annualized return, excess return against a same-day full-market equal-weight reference, long-only Sharpe, maximum drawdown, RankIC, RankIC information ratio, long–short Sharpe, and turnover. The in-sample score combines strategy quality and feature diversity; out-of-sample scoring uses the same metric dimensions with a fixed, unpublished combination rule.Source: §2.4 · pp. 4–5
3 Experiments
Across nine models, iteration raises scores more reliably than sealed-period returns, and no model’s 2024 strategy beats the equal-weight market. The strongest model-level outcome comes from Astra’s cash timing, while Kepler-1 substantially outperforms both the market and the models.
3.2 Performance Across Models and Reasoning Efforts
- Only Sol at medium effort beats the 17.1% market ARR, while 13 of 38 configurations earn positive ARR.Most strategies keep under half their capital in stocks, capturing only part of the market’s gain while paying trading costs.Source: §3.2 · p. 5, Table 2 · p. 6
- The highest-scoring effort is not the highest supported effort for six of nine models.Effort effects reflect discrete design choices: Astra’s xhigh and max runs use a market-state filter and hold cash on 70–75% of days, unlike its low-, medium-, and high-effort runs.Source: §3.2 · pp. 5–6, Table 2 · p. 6
3.3 Why Models Fall Short: Ranking Skill Does Not Reach the Portfolio
- For most models, returns peak in the second or third score decile and decline toward the top, leaving H–L between −10.5 and +4.7 bp.Terra and Astra rise across deciles; average RankIC is almost uncorrelated with ARR across the 38 iterative runs (Spearman 0.09).Source: Figure 3 · p. 7, §3.3 · p. 7
3.4 Case Study: Astra Learns Market Timing
- Astra’s xhigh strategy stays in cash until market conditions turn positive, avoiding the early-2024 decline but ending below the market.It first buys on 28 February; its January–February maximum drawdown is −1.7% versus −29.6% for the market, but it is invested on only 25% of trading days.Source: §3.4 · pp. 7–8, Figure 4 · p. 7
3.5 Iteration Raises Scores More Reliably Than Returns
- Iteration raises total score for all nine models, but sealed-period ARR is higher in iterative runs for only four.Mean score rises from 36.47 to 43.55, while mean ARR gain is 2.9 pp; iterative best configurations lose more score between research and sealed windows than one-shot configurations.Source: §3.5 · p. 8, Table 3 · p. 8
- Research-period progress is concentrated in a few large repairs to weak first candidates, while typical gains remain modest.Without DeepSeek and Sol, the mean final gain is 6.5 pp; at the second candidate the median gain is 6.5 pp, rising to 7.0 pp by the end.Source: Figure 5 · p. 8, §3.5 · pp. 8–9
3.6 Trading Styles
- Most models tilt toward smaller stocks, higher book-to-price, and lower volatility, but style does not separate winners from losers.Astra and Sonnet have similar value and volatility tilts yet finish at 1.11 and 0.88, respectively; earnings-to-price is negative for six models.Source: Figure 6 · p. 9, §3.6 · p. 9
3.7 Kepler-1 Against the Models
- Kepler-1 ends 2024 at 1.93 against 1.17 for the market, while the best model, Astra, ends at 1.11.Only four models finish above starting capital; Kepler-1’s Sharpe ratio is 2.98 against 1.89 for the best model.Source: §3.7 · p. 10, Figure 7 · p. 10
4 Related Work
Prior work explores formulaic alpha generation and varied financial-agent evaluation tasks, but these approaches differ in their research scope and evaluation units. Quant-Arena connects bounded research to the sealed financial outcome of a committed strategy.
4.1 Formulaic Alpha Mining with Language Models
- Formulaic alpha methods search executable expressions or use language models to connect hypotheses, symbolic construction, and empirical diagnosis.These systems move beyond one-shot formula generation, while Quant-Arena evaluates resulting model runs through executable outputs and a frozen final-assessment boundary.Source: §4.1 · p. 10
4.2 Quantitative Benchmarks and Evaluation Protocols
- Financial benchmarks vary in whether they assess code correctness, sequential trading decisions, factor and strategy generation, or agent behavior and information access.Their evaluation units range from code-level outcomes and short repair rounds to repeated decisions, generated implementations, and separated trading-agent behavior.Source: §4.2 · p. 11
- Quant-Arena controls repeated backtest access through frozen data releases, bounded budgets and feedback, date-scoped execution, and sealed assessment of one committed strategy.Its protocol links raw feature construction and bounded research to the sealed financial outcome, complementing code-, factor-, and trading-decision benchmarks.Source: §4.2 · p. 11
5 Conclusion
Quant-Arena evaluates complete quantitative research runs, linking feature construction and candidate revision to a committed strategy’s sealed financial assessment. In the reported results, only one of 38 iterative configurations beats the equal-weight market, and the findings support evaluating both research development and financial outcomes.
5 Conclusion
- Quant-Arena connects raw-market feature construction, candidate revision, and a committed strategy’s sealed financial assessment through shared interfaces and portfolio rules.The shared cash-account evaluator produces comparable simulated outcomes.Source: §5 · p. 11
- Only one of 38 iterative configurations beats the equal-weight market, and the best model score remains below Kepler-1’s score.The best model scores 51.3 against 77.6 for Kepler-1.Source: §5 · p. 11
- The findings support evaluating quantitative research through both its development process and the financial outcome of its committed strategy.Iteration raises overall scores for all nine models but sealed-period ARR for only four, while candidate gains often come early and plateau.Source: §5 · p. 11
A Suite Specification
The suite specification describes a feature limit, supported board-entry modes, and the run records retained by the platform.
A Suite Specification
- A strategy may declare up to 300 named features.Source: A Suite Specification · p. 14
- The public specification is presented as Table 4 for the selected A-share suite.Source: Table 4 · p. 14
- One-shot and iterative entries use the standard harness; system entries use participant-controlled agents, and custom entries support self-hosted models or unsupported endpoints.Source: A Suite Specification · p. 14
- Run records include model and harness details, model calls, research evaluations, the committed strategy, and both window assessments.Failed and cancelled runs remain recorded, and each official leaderboard entry links to its run record.Source: A Suite Specification · p. 14
B.1 One-Shot Results Across Reasoning Efforts
One-shot results are modest, and reasoning effort does not consistently improve scores. Only six configurations have positive sealed-period ARR, and none exceeds the market.
B.1 One-Shot Results Across Reasoning Efforts
- Six of 38 one-shot configurations have positive sealed-period ARR, and none exceeds the 17.1% market.The median configuration scores 31.61, and 4 of 38 exceed 40 points.Source: §B.1 · p. 14
- For eight of nine models, the best one-shot effort is not the highest supported, and effort-related score swings can be large.Terra scores 42.14 with no reasoning and 20.98 at max; Sol scores 32.15 at low and 22.31 at max.Source: §B.1 · p. 14
- Only DeepSeek’s score rises with effort, from 20.48 with no reasoning to 29.92 at max.Each one-shot cell reflects a single draft, with no feedback loop to correct a poor draft.Source: §B.1 · p. 14
B.2 Matched-Effort Comparison
At matched reasoning efforts, iterative runs score higher across all nine models, while ARR improves in fewer comparisons. The score gap is larger because one-shot scores often fall at higher effort.
B.2 Matched-Effort Comparison
- At each model’s highest supported effort, iterative runs score higher for all nine models, with mean scores of 40.88 versus 30.18.Iterative runs also have higher ARR in six of nine pairs.Source: §B.2 · p. 14
- Across 38 matched model–effort cells, iterative runs score higher in 34, have higher ARR in 26, and higher RankIC in 28.For highest-effort matches, Gemini is evaluated at high effort and the other models at max.Source: §B.2 · p. 14
- The matched-effort gap exceeds the best-configuration gap because one-shot scores more often fall with effort, making max effort rarely the best one-shot setting.The one-shot setting receives no performance feedback.Source: §B.2 · p. 15, Table 5 · p. 15
C.1 Signal Similarity and Portfolio Overlap
Models’ daily rankings are moderately correlated, but their selected top-75 stock sets overlap little. Opus–DeepSeek are most similar, while Terra is most distinct.
C.1 Signal Similarity and Portfolio Overlap
- Mean pairwise rank correlation is 0.47, but mean Top-75 Jaccard overlap is 6.1%, with a maximum of 35.6%.Figure 8 compares pairwise mean daily Spearman signal correlations and Top-75 Jaccard overlaps across 36 model pairs.Source: Figure 8 · p. 16, §C.1 · p. 16
- Opus and DeepSeek are the most similar pair, with correlation 0.94 and overlap 35.6%.Sol–GLM correlate at 0.87, and Sonnet–Astra at 0.84.Source: §C.1 · p. 16
- Terra is the most distinct strategy, with mean correlation 0.14 against the other eight models.Opus and DeepSeek correlate at about 0.5 or less with every other model, while remaining models largely form one cluster.Source: §C.1 · p. 16