Source-linked AI summary
QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
Alexey Khoroshilov, Alexey Chernysh, Orkhan Ekhtibarov, Nini Kamkia, Dmitry Zmitrovich
TL;DR
LLM generation of executable trading strategies remains underbenchmarked because success requires domain logic, specialized API use, observable trades, and semantic fidelity. QuantCode-Bench evaluates these requirements across 400 Backtrader tasks in single-turn and agentic settings, finding that iterative feedback raises the best models’ Judge Pass from roughly three quarters to 95–98%.
Problem
Existing code benchmarks do not adequately measure whether LLMs translate natural-language trading ideas into executable and semantically faithful strategies.
Method
QuantCode-Bench evaluates 400 Backtrader strategy-generation tasks through nested syntax, execution, trade, and LLM-judge stages in single-turn and feedback-driven multi-turn settings.
Results
The best single-turn Judge Pass reaches roughly three quarters of the benchmark, while iterative feedback raises the best models to 95–98%.
Takeaways & Limitations
The main challenge is operationalizing trading logic and task semantics, while interactive debugging repairs a substantial fraction of errors.
Takeaways & Limitations
Evaluation uses only the Backtrader framework and an LLM judge, limiting transferability and providing no absolute guarantee of semantic correctness.
Abstract
from arXiv · showhide
Large language models have demonstrated strong performance on general-purpose programming tasks, yet their ability to generate executable algorithmic trading strategies remains underexplored. Unlike standard code benchmarks, trading-strategy generation requires simultaneous mastery of domain-specific financial logic, knowledge of a specialized API, and the ability to produce code that is not only syntactically correct but also leads to actual trades on historical data. In this work, we present QuantCode-Bench, a benchmark for the systematic evaluation of modern LLMs in generating strategies for the Backtrader framework from textual descriptions in English. The benchmark contains 400 tasks of varying difficulty collected from Reddit, TradingView, StackExchange, GitHub, and synthetic sources. Evaluation is conducted through a multi-stage pipeline that checks syntactic correctness, successful backtest execution, the presence of trades, and semantic alignment with the task description using an LLM judge. We compare state-of-the-art models in two settings: single-turn, where the strategy must be generated correctly on the first attempt, and agentic multi-turn, where the model receives iterative feedback and may repair its errors. We analyze the failure modes across different stages of the pipeline and show that the main limitations of current models are not related to syntax, but rather to the correct operationalization of trading logic, proper API usage, and adherence to task semantics. These findings suggest that trading strategy generation constitutes a distinct class of domain-specific code generation tasks in which success requires not only technical correctness, but also alignment between natural-language descriptions, financial logic, and the observable behavior of the strategy on data.
1 Introduction
QuantCode-Bench addresses the lack of domain-specific benchmarks for generating executable trading strategies by evaluating both technical execution and semantic fidelity. Its results show that current models struggle mainly with operationalizing trading logic rather than producing syntactically valid code.
- Motivation: Trading-strategy generation requires interpreting financial logic, using Backtrader correctly, and producing trades that reflect the description.Models must handle indicators, entry and exit conditions, position management, framework interfaces, and execution semantics.
- Motivation: Unlike standard code benchmarks, executable and compiling strategies may still generate no trades or implement the wrong task.The benchmark therefore requires layered validation beyond compilation and backtest completion.
- Benchmark: QuantCode-Bench contains 400 Backtrader tasks and evaluates both first-attempt generation and iterative repair with structured feedback.The two settings separate one-shot generation ability from interactive error-repair capability.
- Evaluation: Its four-stage pipeline checks syntax, historical-data execution, trade presence, and semantic alignment judged by an LLM.These stages distinguish code correctness, executable strategy construction, trading-signal generation, and faithful implementation of the requested idea.
- Findings: Single-turn frontier models nearly compile perfectly, yet their best Judge Pass remains roughly half the benchmark.The largest losses arise from operationalizing trading logic rather than surface-level syntax.
- Contributions: The work contributes a reproducible benchmark, multi-level evaluation framework, model comparison, and failure-mode analysis for domain-specific code generation.The benchmark is released as a foundation for future research.
2 QuantCode-Bench
QuantCode-Bench is a 400-task English dataset for generating Backtrader trading strategies from descriptions of varying formality, structure, and difficulty. Its design uses structural enrichment and a deliberately nontrivial framework API to test realistic domain-specific code generation.
- Task Definition: QuantCode-Bench evaluates whether generated Backtrader strategies compile, execute, trade, and match the described trading idea.These are four nested requirements, with each stage strengthening the definition of success.
- Dataset: The dataset contains 400 trading-strategy generation tasks collected from sources differing in formality, structure, and detail.The supplied passages identify the size and heterogeneous source character, while the tables summarize source and difficulty distributions.
- Dataset: Each task is structurally enriched with indicators, entry and exit conditions, additional rules, and an easy, medium, or hard difficulty category.Rules may be extracted from explicit or implicit descriptions.
- Resources: The benchmark uses English descriptions and releases its code, dataset, and materials through a public repository and benchmark page.The passages provide the project URLs.
- Framework: Backtrader was selected because its indicators, data lines, order execution methods, and indexing conventions create nontrivial API demands.This complexity reduces success through superficial reproduction of standard templates.
3 Evaluation Methodology
The evaluation methodology treats trading-strategy success as a sequence of four nested checks and uses semantic judging to distinguish functional execution from faithful task implementation. It compares one-shot generation with iterative repair using structured diagnostic feedback.
- Four-stage pipeline: A strategy succeeds only after passing Compilation, Backtest, Trade, and Judge stages sequentially.The stages assess syntax, runtime execution on historical data, at least one trade, and semantic agreement with the description.
- Four-stage pipeline: The staged pipeline localizes failures among syntax, execution, signal generation, and incorrect trading logic.A single aggregate score would obscure these qualitatively different failure causes.
- Semantic validation: The LLM judge verifies indicator correspondence, entry and exit behavior, and relevance to the requested task rather than a generic template.Semantic validation catches technically functional but substantively incorrect strategies.
- Interaction settings: In single-turn evaluation, the model must generate a correct strategy on its first attempt.This measures initial knowledge of the domain, library, and common strategy templates.
- Interaction settings: In agentic multi-turn evaluation, structured error feedback enables code revision and retries for up to 10 attempts.This setting measures iterative repair, local search, and use of diagnostic information.
4 Results
Results show a sharp gap between strong compilation performance and weaker execution, trading, and semantic compliance in single-turn generation. Iterative feedback improves final success, especially for models whose errors are locally repairable.
- Single-turn Results: Frontier models are almost uniformly strong on compilation but substantially more dispersed on later evaluation stages.The ranking reveals that model differences emerge after syntactic correctness.
- Single-turn Results: Single-turn compilation has nearly ceased to be a bottleneck, while major quality losses occur at the Backtest and Trade stages.A high Compilation Rate does not automatically produce a high Judge Pass.
- Agentic Results: The agentic protocol measures how quickly models convert partial failures into final task success across cumulative turns.Table 4 reports this cumulative multi-turn performance.
- Agentic Results: Iterative feedback is especially effective for strong models because a substantial fraction of their errors are locally repairable within a small number of attempts.The remaining differences arise mainly in execution, trade generation, and semantic compliance rather than syntax.
5 Error Analysis
Error analysis shows that single-turn failures concentrate after compilation, especially in backtesting and trade generation, while agentic repair reduces technical errors but leaves persistent semantic and logic-level failures.
- Compilation has almost ceased to be the main single-turn failure point; difficulty concentrates in Backtest and No trades stages.
- The most frequent late-stage failure comprises strategies that compile and backtest successfully but place no trades on the data.Common causes include overly strict entry conditions, insufficient historical context, and incorrect indicator operationalization.
- The second most frequent failure category is __bool__ / Line object errors caused by incorrect handling of Backtrader line objects in boolean conditions.
- Agentic feedback shifts unresolved failures toward task mismatch, inactive signal conditions, and __bool__ / Line object errors, while missing attribute/method errors remain comparatively infrequent.After 10 attempts, Judge rejection becomes one of the main causes of final failure.
- Iterative debugging is primarily effective for technical repairs and is substantially less effective when models misunderstand the natural-language task.
6 Discussion
The discussion argues that trading-strategy generation tests operational and semantic capabilities beyond syntax. Iterative feedback helps repair many local errors, but specification ambiguity and semantic validation remain central challenges.
- Modern LLMs have largely solved surface-level syntactic generation, shifting the challenge to executable, data-activatable, and semantically correct operational formalization.
- Large improvements under iterative feedback show that many errors are locally repairable specification or API violations rather than fundamental inability to generate strategies.
- Vague conversational descriptions are often harder for models than more formal parameterized formulations, even when the latter are conceptually more complex.
- Programming specialization does not guarantee superiority in domain-specific strategy generation.The task also requires interpreting financial intent, translating descriptions into meaningful behavior, and selecting realistic trigger conditions.
- Without semantic validation, Trade Rate systematically overestimates success because technically passing strategies may not match the requested task.Including the Judge stage makes semantic compliance part of the reward objective and prevents repeated use of unrelated working templates.
7 Limitations
QuantCode-Bench’s scope and evaluation design impose important boundaries: it uses only Backtrader, relies on an LLM judge for semantic validation, and does not assess investment quality.
- Framework scope: Backtrader is the sole framework and execution environment, limiting transferability to other algorithmic-trading libraries and environments.The authors identify QuantConnect/LEAN and Zipline as natural extensions.
- Evaluation scope: Semantic evaluation relies on an LLM judge that may overlook subtle logic mismatches and exhibit positional, stylistic, or model-specific biases.The judge strengthens evaluation beyond purely technical validation but cannot guarantee absolute semantic correctness.
- Economic scope: The benchmark does not evaluate profitability, risk robustness, or the economic quality of generated strategies.Its stated focus is executable strategy generation from descriptions rather than investment effectiveness.
8 Conclusion
QuantCode-Bench evaluates whether LLMs translate natural-language trading ideas into executable, behaviorally valid implementations. Iterative feedback substantially improves results, while one-shot performance remains incomplete and the benchmark supports future domain-specific code-generation research.
- Conclusion: QuantCode-Bench formalizes trading-strategy generation as nested checks for syntax, execution, trades, and semantic alignment.This structure evaluates behavioral validity in addition to surface-level code quality.
- Conclusion: The maximum single-turn Judge Pass reaches roughly three quarters of the benchmark, while agentic feedback raises the best models to 95–98%.The authors interpret this improvement as evidence that many errors are repairable through interactive debugging.
- Conclusion: Trading-strategy generation requires specialized API command, executable code, realistic trading logic, and adherence to natural-language semantics.Modern models perform well on syntax and basic infrastructure but remain limited in robust one-shot formalization and precise strategy implementation.
- Conclusion: QuantCode-Bench can support future research on domain-specific code generation, agentic software repair, and LLM evaluation in finance.
A Example Tasks from QuantCode-Bench
The appendix presents representative QuantCode-Bench tasks spanning the benchmark’s difficulty levels. These examples illustrate how task specifications are organized for evaluation.
- Example Tasks from QuantCode-Bench: The appendix presents one representative example for each difficulty level in QuantCode-Bench.
- Example Tasks from QuantCode-Bench: The examples are presented as task specifications rather than as aggregate benchmark results.
- Example Tasks from QuantCode-Bench: The examples provide concrete instances of the benchmark’s evaluated tasks.
A.1 Easy example
The appendix illustrates easy, medium, and hard trading tasks with increasingly detailed specifications for signals, risk controls, filters, and execution constraints.
- A.1 Easy example: The easy SPY task uses a calendar-based monthly strategy on daily data without technical indicators.It enters on the first trading day, exits on the last trading day, invests 100% of available capital, and reports initial and final portfolio value.
- A.1 Easy example: The SOFI entry rules include pullbacks to gap-fill levels or rising intraday EMAs followed by upward reversal, or breakouts from 15-minute flags.
- A.1 Easy example: The hard SNAP task is a one-minute scalping strategy combining Stochastic crossovers, RSI(3), volume liquidity, and ATR(14) volatility filters.Long and short entries use mirrored thresholds, including volume above 1.2 × SMA(volume, 20) and ATR(14) ≥0.03.
- A.1 Easy example: The SNAP specification adds fixed take-profit and stop-loss distances, reversal or three-bar timeout exits, session restrictions, and 100–500-share position limits.An optional $25,000 balance filter may be applied for PDT compliance.