Source-linked AI summary
QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code
Alexey Chernysh, Orkhan Ekhtibarov, Dmitry Zmitrovich
TL;DR
Executable trading-code generation requires more than plausible text: strategies must run and implement the request. This paper evaluates framework-code continued pretraining and validated SFT, finding gains in strategy success alongside a tool-call retention failure that recovery SFT only partly addresses.
Problem
Trading-code models must translate strategy requests into framework programs that execute, produce trades, and remain semantically faithful to the request.
Source: §1 · p. 1Method
The study evaluates framework-oriented continued pretraining and SFT, alongside repository-level tasks and tool-call recovery checks.
Source: §1 · p. 3, §3.6 · p. 7, §3.7 · p. 7Results
58.2% Judge Pass after SFT on Qwen3.6-35B-A3B; specialization also degraded parser-conformant calls, and recovery SFT restored formatting but not base-level repository resolution.
Source: §7 · p. 14Takeaways & Limitations
Specialized code generation should be trained and evaluated as behavior in an environment, with capability-retention checks when deployment depends on structured tool use.
Source: §7 · p. 15The evaluation does not measure profitability, market-regime robustness, transaction costs, market impact, or production risk, so success does not establish profitability or deployment suitability.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Large language models are strong general-purpose code generators, but executable algorithmic trading remains a demanding specialization target: a model must translate a natural-language strategy specification into correct program logic for a specialized trading framework, execute on historical data, produce trades, and remain semantically faithful to the request. We study two complementary mechanisms for specializing language models for this setting: continued pretraining on algorithmic-trading framework code and supervised fine-tuning (SFT) on agent-validated request-to-code pairs. Evaluation is centered on QuantCode-Bench, our 400-task benchmark for Backtrader strategy generation, together with a repository-level SWE-bench-like track. Continued pretraining improves single-turn Judge Pass from 41.5% to 47.5% for Qwen3.5-397B-A17B and from 27.8% to 33.0% for Qwen3.6-35B-A3B. SFT applied after continued pretraining yields a larger gain for Qwen3.6-35B-A3B, reaching 58.2% Judge Pass and 83.5% successful backtests; in agentic evaluation it raises first-turn success from 22.3% to 58.3% and final success after up to 10 turns from 47.5% to 79.5%. Continued pretraining alone improves first-turn agentic success but lowers final success after repair from 47.5% to 32.5%, consistent with degraded instruction following, whereas SFT improves both. We also identify a capability-retention failure: domain specialization degrades parser-conformant structured tool calling, and targeted recovery SFT restores tool-call formatting but not the base checkpoint's repository-level agent performance. The results show that framework-oriented pretraining, validated SFT, and explicit capability-retention evaluation address distinct failure modes in domain-specific executable code generation.
1 Introduction
Executable trading code must satisfy framework, runtime, trading-behavior, and semantic requirements that text-based correctness alone cannot capture. The study compares domain pretraining and validated SFT, finding different effects on strategy generation and iterative repair, alongside a tool-call retention failure.
1 Introduction
- Trading requests require framework-aware implementation of indicators, entry and exit rules, sizing, stops, and risk controls, with runtime behavior and fidelity to the requested strategy both at stake.Code may run yet generate no trades or implement a different strategy, failures that text similarity or compilation accuracy alone cannot reliably capture.Source: §1 · pp. 1–2
- QuantCode-Bench evaluates 400 Backtrader tasks through executable checks for validity, successful execution, at least one trade, and semantic alignment, with single-turn and up-to-10-attempt agentic settings.The agentic setting provides structured tool feedback between attempts.Source: §1 · p. 2
- Continued pretraining targets framework and API familiarity, whereas SFT directly maps realistic requests to validated implementations; evaluating both in sequential lineages distinguishes their effects.The SFT examples are static targets, not iterative-repair trajectories.Source: §1 · p. 2
- SFT substantially improves strategy-generation and agentic success, while continued pretraining alone raises first-turn success but lowers final success after iterative repair below the base checkpoint.The authors interpret the repair result as consistent with degraded processing of conversational feedback.Source: §1 · p. 2
- A repository-level track extends evaluation to real trading repositories, where models inspect issues, patch code, run tests, and repair from tool feedback.It probes transfer from isolated strategy generation to longer-horizon software engineering.Source: §1 · p. 2
- Domain specialization can degrade structured tool-call formatting required by the interactive agent harness, a capability not directly measured by QuantCode-Bench.The report evaluates weight merging and dedicated tool-use SFT, distinguishing parser-conformant calls from downstream tool-use quality.Source: §1 · pp. 2–3
2 Related Work
The related work places this study at the intersection of domain-specific code training, agentic repository engineering, and financial-domain evaluation. It distinguishes executable trading-code competence from general coding or financial reasoning.
2 Related Work
- Domain-adaptive pretraining and code-centric models [5] [6] [7] [8] specialize general models toward target distributions, while this work narrows the target to trading-framework APIs and conventions.Its aim is framework familiarity for executable strategies, not merely stronger general coding performance.Source: §2 · p. 3
- The study uses Qwen-family checkpoints rather than proposing a foundation architecture, evaluating a narrowly defined executable financial-software domain amid work on long-horizon agentic engineering [3] [4].The Qwen technical report describes dense and mixture-of-experts models across scales; GLM-5 also emphasizes agentic engineering.Source: §2 · p. 3
- SWE-bench [9] frames software engineering as repository-level issue resolution involving code inspection, patching, and executable tests; this work adopts that paradigm for algorithmic-trading repositories.The task domain changes, while verification remains test-based.Source: §2 · p. 3
- Financial-domain evaluation such as FINESSE-Bench [2] targets financial knowledge and technical-analysis reasoning, whereas this report targets faithful implementation of trading requests as executable code.Financial knowledge and correct framework implementation are treated as distinct capabilities.Source: §2 · p. 3
3 Methods
The methods define success through executable strategy behavior and compare sequential training stages using framework-code pretraining and validated request-to-code SFT. Evaluation spans standalone strategies, repository tasks, and recovery of structured tool calling.
3.1 Task formulation
- A strategy maps a natural-language request to a framework program and succeeds only if it executes, produces a trade, and passes semantic alignment; profitability is outside the correctness objective.Compilation is also tracked, but the reported executable conditions require successful backtesting, a trade, and judge approval.Source: §3.1 · p. 4
3.2 Experimental structure and comparability
- The comparable lineages are Qwen3.6-35B-A3B base → continued pretraining → SFT and Qwen3.5-397B-A17B base → continued pretraining.This permits stage-by-stage comparisons within each lineage; Table 1 summarizes the training and evaluation regimes.Source: §3.2 · pp. 4–5, Table 1 · p. 5
3.3 Continued pretraining on algorithmic-trading code
- Continued pretraining uses trading-framework and repository code to build API familiarity through domain-code continuation, without an explicit user-request-to-answer mapping.The corpus contains approximately 5–6B tokens; results are reported as internal training snapshots rather than a general pretraining recipe.Source: §3.3 · pp. 5–6
3.4 Supervised fine-tuning data
- SFT maps strategy requests to complete Backtrader implementations, with agent-generated targets using native framework abstractions and validation for execution defects; its novelty centers on target construction and validation.Requests draw on realistic trading material and exclude QuantCode-Bench tasks from the SFT pool.Source: §3.4 · p. 6
3.5 QuantCode-Bench evaluation
- QuantCode-Bench evaluates 400 tasks in single-turn and agentic settings, tracking backtests, trades, and semantic-judge success; agentic attempts may use feedback for up to 10 turns.Generations use greedy decoding, and the principal comparison reports T1 and T10.Source: §3.5 · pp. 6–7
3.6 Repository-level algorithmic-trading track
- The repository track contains approximately 200 verified trading-repository tasks, but specialized checkpoints could not be evaluated because the track requires parser-conformant tool calls.Success is determined by packaged repository tests; Backtrader backtests are not part of this benchmark.Source: §3.6 · p. 7
3.7 Tool-call recovery fine-tuning
- Tool-call recovery combines a 0.65-specialized/0.35-original weight merge with recovery SFT initialized from the merged checkpoint, trained on general and trading agent trajectories.The SFT supervises assistant turns, including intermediate tool calls, with 16k- and 32k-token sequence-length variants.Source: §3.7 · pp. 7–8
- Recovery checks assess parser-conformant formatting qualitatively and repository-task resolution quantitatively; no standalone parse-success rate was retained.Measurements were made with reasoning disabled.Source: §3.7 · p. 8
4 Results
Continued pretraining improves single-turn outcomes, while subsequent SFT adds larger single-turn and agentic gains. Domain specialization also exposes tool-call compatibility and repository-agent limitations.
4.1 Continued pretraining improves single-turn Judge Pass
- Judge Pass rose from 41.5% to 47.5% for Qwen3.5-397B-A17B and from 27.8% to 33.0% for Qwen3.6-35B-A3B after continued pretraining.Source: Table 2 · p. 8, §4.1 · p. 8
4.2 SFT produces the largest observed single-turn improvement
- 58.2% Judge Pass and 83.5% Backtest were reached by Qwen3.6-35B-A3B after SFT, with gains spanning runnable strategies, trade generation, and semantic success.Against the domain-pretrained checkpoint, Backtest rose 29.7 points, Trades 23.2 points, and Judge Pass 25.2 points.Source: §4.2 · p. 9
4.3 SFT strongly improves first-turn and final agentic success
- T1 success rose from 22.3% to 58.3% and T10 success from 47.5% to 79.5% across the Qwen3.6-35B-A3B base-to-SFT progression.Source: Table 4 · p. 10, §4.3 · p. 10, Figure 2 · p. 10
- The domain-pretrained checkpoint improved T1 from 22.3% to 32.3% but reached only 32.5% at T10, 15.0 points below the base checkpoint.Source: §4.3 · p. 10
4.4 Domain specialization degrades structured tool-call generation
- QuantCode-Bench does not require native tool-call syntax, and malformed calls prevented specialized checkpoints from meaningful repository-track interaction.Source: §4.4 · p. 11
- A 0.65/0.35 parameter merge did not restore parser-conformant tool calling, but this result does not rule out other merge methods or coefficients.Source: §4.4 · p. 11
- Recovery SFT restored parser-accepted formatting qualitatively, but repository solves remained 16.3% for the 16k variant and 13.0% for the 32k variant, below the base's 23.7%.The table measures downstream agent utility, not formatting recovery; no standalone parse-success rate was retained.Source: §4.4 · pp. 11–12
5 Discussion
The discussion finds that continued pretraining and SFT improve different parts of executable strategy generation, while also exposing capability-retention and repository-level challenges.
5 Discussion
- Within the directly comparable Qwen3.6-35B-A3B lineage, continued pretraining and subsequent SFT improve different parts of executable strategy generation; separately, Qwen3.5-397B-A17B base-to-pretraining comparison shows a gain at larger scale.The pretraining gain is smaller than the SFT gain, which directly pairs strategy intent with executable target code.Source: §5 · p. 12
- Continued pretraining yields 5.2- and 6.0-point Judge Pass gains at 35B and 397B scales, supporting framework knowledge as a meaningful bottleneck.The intervention exposes models to repository code and usage patterns rather than explicit benchmark solutions.Source: §5 · p. 12
- SFT directly pairs strategy intent with executable code, and validation suppresses targets that fail in the execution environment.This targets semantic translation gaps that domain-code exposure alone does not directly solve.Source: §5 · p. 12
- Raising T1 from 22.3% to 58.3% improves the initial action distribution, reducing expected model calls, backtests, and repair iterations.The discussion frames this as a latency and compute improvement for interactive coding assistants.Source: §5 · pp. 12–13
- The SFT checkpoint still gains more than 20 percentage points from T1 to T10, so specialization does not eliminate the value of tools.Many residual defects remain repairable after execution.Source: §5 · p. 13
- Repository-level engineering remains difficult because it requires code discovery, project-specific understanding, interface preservation, and test-driven repair.Low absolute solve rates leave this setting open and motivate repository-level SFT trajectories.Source: §5 · p. 13
- Specialization can degrade parser-conformant tool calls; recovery SFT restored formatting, but repository-level performance remained below the base checkpoint.Restoring valid syntax is not equivalent to restoring decisions about which tools to call and how to use their results.Source: §5 · p. 13
- QuantCode-Bench separates compilation, execution, trade generation, and semantic alignment, revealing distinct capability shifts across training stages.The tool-call regression was not directly measured by the primary strategy-generation evaluations and surfaced in the target agent harness.Source: §5 · p. 13
6 Limitations
The reported conclusions are bounded by the training lineages and benchmark snapshot, evaluation proxies, unmeasured deployment and profitability properties, and repository-track scope.
6 Limitations
- SFT was tested only in the 35B lineage, and its effect without prior continued pretraining was not measured.The compared checkpoints form sequential lineages in the current QuantCode-Bench snapshot.Source: §6 · p. 14
- The report does not claim a complete public reproduction package for every historical run from the evolving training program.Future releases should freeze data and framework versions, optimizer settings, random seeds, and schedules.Source: §6 · p. 14
- The LLM judge is an automated semantic proxy, so it can miss subtle trading-logic mismatches or exhibit model-specific biases [1].Executable checks precede judging but do not remove semantic-evaluation uncertainty.Source: §6 · p. 14
- Neither QuantCode-Bench nor SFT validation measures profitability, market-regime robustness, realistic transaction costs, market impact, or production risk.Success rates concern faithful implementation of requested code, not strategy profitability or deployment suitability.Source: §6 · p. 14
- Continued pretraining on public repository code raises contamination concerns, including possible overlap with benchmark domains and issue/patch leakage.The report calls for stronger decontamination and exact-match filtering.Source: §6 · p. 14
- Tool-call recovery is reported qualitatively because the internal snapshot retained no standalone parser-validity rate.The report recommends measuring parse success, argument validity, and downstream task success separately.Source: §6 · p. 14
- The repository track contains approximately 200 verified tasks and remains under development, limiting broader characterization of algorithmic-trading software engineering.Broader repositories, frameworks, languages, and market-data environments are needed.Source: §6 · p. 14
7 Conclusion
Domain-specific training improves algorithmic-trading code generation, while specialization also exposes a retained-capability failure in structured tool use. The conclusion calls for evaluating executable code in its environment and checking retained capabilities.
7 Conclusion
- 58.3% first-turn agentic success and 79.5% success after up to 10 turns for the final Qwen3.6-35B-A3B checkpoint, versus 22.3% and 47.5% for base.Source: §7 · p. 14
- Continued pretraining raised Judge Pass from 41.5% to 47.5% for Qwen3.5-397B-A17B and from 27.8% to 33.0% for Qwen3.6-35B-A3B.Source: §7 · p. 14
- Recovery SFT restored operational tool-call formatting, but repository-level task resolution remained below the Qwen3.6-35B-A3B base checkpoint.Source: §7 · p. 14
- Framework-oriented pretraining improves software-ecosystem knowledge, validated SFT maps user intent to executable code, and structured-tool deployments need capability-retention checks.Source: §7 · p. 15
- Next steps include testing more model scales, repository-level repair trajectories, separate tool-call validity and use-quality measures, and real user request and correction trajectories.Source: §7 · p. 15