Source-linked AI summary
MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code
Siyu Wang, Yifan Wang, Yuecheng He
TL;DR
Existing code benchmarks test functional correctness against unit tests, while finance benchmarks test forecasting; neither measures whether code implements the requested strategy behavior. MintEval generates reference strategies from composable blocks, back-translates them into colloquial instructions, and compares generated and reference actions bar by bar under identical data and frictions. Among 1,767 compiled runs whose specification was fully correct, 0.792 had ActionMatch below 0.9.
Problem
Existing code benchmarks test functional correctness against unit tests, while finance benchmarks test forecasting; neither measures whether code implements the requested strategy behavior.
Source: §1 · p. 1Method
Results
Among 1,767 compiled runs whose specification was fully correct, 0.792 had ActionMatch below 0.9.
Source: §5 · p. 3Takeaways & Limitations
Correctly reading a strategy specification does not ensure its persistent logic is implemented faithfully.
Source: §5 · p. 3The frontier-model result covers one model on a stratified subset of 200 tasks; the authors call for harder tasks and frontier models from other families.
Source: §5 · pp. 2–3
Abstract
from arXiv · showhide
Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: generated code runs, a backtest plots, yet the risk logic that the trader described is not the logic being executed. Existing code benchmarks test functional correctness on unit tests and finance benchmarks test forecasting; neither measures whether an implementation behaves like the strategy that was asked for. We introduce MintEval, a benchmark in which reference strategies are generated programmatically from a library of composable building blocks, back-translated into colloquial trader instructions, and re-implemented by the model under test. Generated and reference programs are executed bar by bar on identical market data and frictions, and compared on their actions rather than on code similarity or profit: alpha is differenced away. MintEval v0 contains 800 tasks on BTCUSDT 15-minute data, stratified by an execution-measured state-span complexity tau that is decoupled from description length. Low-cost models reach a mean ActionMatch of at most 0.544 and reproduce at most 0.087 of tasks exactly; on a stratified subset of 200 tasks a frontier model (Claude Opus 5.5) reaches 0.889 and reproduces 0.575 exactly, yet still fails silently on 0.275 of tasks. Given a menu of building blocks, models identify the strategy almost perfectly, yet 79.2% of the implementations whose specification was read correctly diverge on more than 10% of active bars. The LLM judge of a recent strategy-generation benchmark, applied verbatim, accepts every one of these silent failures.
1 Introduction
MintEval addresses a gap between testing whether code runs and whether it implements a trader’s intent. It evaluates strategy implementation through bar-level behavioral equivalence with a reference program.
1 Introduction
- Trading-code errors can run silently: a plausible backtest may conceal risk logic that differs from the trader’s intent.Examples include a breakeven stop that never moves or a cooldown that resets every bar.Source: §1 · p. 1
- Operational trading-code failures are costly, and backtests are already a fragile basis for decisions.Source: §1 · p. 1
- Unit-test code benchmarks use precise human-written specifications, while trading strategies are stateful, sensitive to mistimed actions, and described colloquially.Persistent state can span thousands of steps, and one mistimed exit can change later decisions.Source: §1 · p. 1
- MintEval scores a task by bar-level behavioral equivalence between a generated strategy and its reference under identical data and frictions.Its contributions also include ActionMatch, contamination-free tasks with separate complexity axes, and open and closed model evaluations.Source: §1 · p. 1
2 Task
A MintEval task pairs a colloquial strategy instruction with a reference program and fixed market data and frictions. Programs are compared by the positions they produce bar by bar, not by their code structure.
2 Task
- Behavioral equivalence requires matching quantized target positions on every bar under the task’s fixed data and frictions.Different program structures can be equivalent if their actions match; a one-operator difference can make them nonequivalent.Source: §2 · p. 1
- Both programs are called at each bar close and return a target position, with optional stop and take-profit levels for the next bar.The model receives the instruction and an interface document describing engine semantics, indicators, and desk conventions.Source: §2 · p. 1
- The engine enforces causality by revealing bars and indicators incrementally and raising an unswallowable error on access to future-bar data.Generated code runs in a subprocess sandbox with limited imports, no I/O, and a 10-second budget.Source: §2 · p. 1
3 Metric Suite
MintEval measures implementation fidelity through action and trade comparisons, return differences, and execution status. ActionMatch focuses on active bars so flat periods do not inflate agreement.
3 Metric Suite
- ActionMatch is the fraction of active bars on which the programs’ quantized targets match.A bar is active when either program holds or targets a non-zero position.Source: §3 · p. 1
- TradeF1 requires exact round-trip matches, while |ES| measures the absolute difference in total return in basis points of initial equity.The absolute return difference prevents profitable implementation errors from counting as faithful and avoids signed errors canceling across tasks.Source: §3 · p. 2
- Runs that raise, time out, or look ahead receive CompileOK = 0, with other metrics left missing rather than imputed.Source: §3 · p. 2
- A silent failure is a compiled run with ActionMatch below 0.9; the authors note that this threshold is a convention, not a cliff.The silent-failure share varies across models and settings when the threshold changes to 0.8 or 0.99.Source: §3 · p. 2
4 Benchmark Construction
MintEval constructs composable strategy tasks and measures complexity through executed state span as well as program description length. The benchmark uses stratified task selection and colloquial back-translations, while its authors identify reference realism as a boundary for finance-facing interpretation.
4 Benchmark Construction
- The library combines signal, direction, filter, sizing, and one to three risk layers, including stateful combinations such as breakout-then-retest systems.It contains 6 signal, 2 filter, 3 sizing, and 9 risk blocks.Source: §4 · p. 2
- Program complexity is represented by description length K and execution-measured state-span statistics τmax and τp90, with R counting cross-bar-read keys.Across the task set, corr(log τmax, K) = -0.03; some block parameters vary τ, enabling within-block estimates.Source: §4 · p. 2
- To address confounding because short-state-span programs trade more often and pay more fees, researchers jointly stratified by τmax quintile and reference-return band, drawing 32 tasks per band for 800 total.Median reference returns across the five quintiles were -0.37, -0.37, -0.37, -0.34, and -0.34.Source: §4 · p. 2
- Reference programs mostly lose after costs, but action-based comparisons do not depend on whether a reference strategy is profitable.The median reference loses 35.1% over two years and 8.9% are profitable after costs; with zero fees and slippage, the median is -0.6% and 47.8% are profitable.Source: §4 · p. 2
- The authors identify reference realism as a limitation: version 1 aims to lower frictions to 2–3 bp and add one-hour bars.They note that more realistic reference candidates would be more convincing to a finance audience.Source: §4 · p. 2
- An LLM translates each generated program into a 40–120-word desk message, then a separate reader checks whether the block-level specification can be recovered.796 of 800 messages pass the gates; 780 are round-trip verified and 20 unverifiable because the reader failed.Source: §4 · p. 2
- Tasks are sampled from a combinatorial space at build time, and neither their instructions nor reference programs exist on the web.Source: §4 · p. 2
5 Preliminary Results
Low-cost models often compile yet diverge behaviorally; a frontier model improves results on a 200-task subset but still has silent failures. Correctly recovering the specification does not ensure correct implementation, and errors persist as state span grows.
5 Preliminary Results
- The evaluation uses GPT-5.4-mini, Claude Haiku, and Qwen2.5-Coder 32B and 7B, with one sample per task and setting.Sampling temperature is 0.Source: §5 · p. 2
- ActionMatch reaches at most 0.544 for low-cost models, which reproduce at most 0.087 of tasks exactly; median |ES| is 442 bp of initial equity.The leaderboard covers 800 tasks; ActionMatch, TradeF1, and |ES| are measured over compiled runs.Source: §5 · p. 2, Table 1 · p. 3
- On the 200-task subset, Claude Opus 5.5 reaches ActionMatch 0.889 and exact reproduction 0.575, yet 0.275 of its tasks remain silent failures.GPT-5.4-mini reaches ActionMatch 0.575 and exact reproduction 0.085 on the same tasks.Source: §5 · p. 2, Table 2 · p. 3
- The v0 limitation notes a need for harder tasks and frontier models from other families.The paper says these are needed for v1.Source: §5 · p. 3
- With a block menu, models recover specifications at SpecMatch 0.935–0.946, but 0.792 of fully correct specifications still have ActionMatch below 0.9.Closed-setting ActionMatch improves only modestly over open-setting performance, for example 0.544 →0.582.Source: §5 · p. 3, Figure 2 · p. 3
- ActionMatch declines with state span: the fully controlled estimate is -0.033 (p=0.049) in the open setting and -0.063 (p=3.7e−05) in the closed setting.The open-setting estimate is described as borderline; the within-block effect is smaller than the raw gradient but does not vanish.Source: §5 · p. 3
- Inspected silent failures include breakout windows containing the current bar, stops re-anchored to each bar’s close, and higher-timeframe filters evaluated on the wrong candle.Without physical causality enforcement, higher-timeframe look-ahead can silently leak future data into a backtest.Source: §5 · p. 3
A code-reading judge does not see silent failures
Applied verbatim to compiled runs, the QuantCode-Bench judge accepted every judged silent failure, though the benchmark gap is one between reading code and checking executed behavior. Changing the state rule shifted compile failures without materially changing compiled-run ActionMatch.
A code-reading judge does not see silent failures
- The judge accepted all 55 Opus 5.5 silent failures (0.275 of its passes) and all 120 GPT-5.4-mini silent failures (0.736 of its passes).The comparison is scoped by Table 3 to the QuantCode-Bench judge versus behavioural equivalence at ActionMatch ≥0.9; 26 GPT-5.4-mini runs could not be judged because of an API budget limit.Source: A code-reading judge does not see… · p. 3, Table 3 · p. 4
- The judge rejected 7 of 7 negative-control pairs with code from a different signal family, but could not detect the right strategy implemented wrongly.It detects the wrong strategy; it cannot detect the right strategy implemented wrongly.Source: A code-reading judge does not see… · p. 3
- Behavioural equivalence addresses evaluations whose final check reads code rather than executing it against a reference.This is the gap the benchmark is designed to close.Source: A code-reading judge does not see… · p. 4
- Under a lenient state rule, CompileOK rose for Qwen-32B open from 0.601 to 0.706 and Qwen-7B closed from 0.488 to 0.713, while Qwen-32B open ActionMatch stayed essentially unchanged at 0.172 vs. 0.169.The rescued programs were mostly silent failures, so the strict rule shifted failures between loud and silent categories without driving the conclusions.Source: A code-reading judge does not see… · p. 4
6 Related Work
MintEval measures whether generated strategy code behaves like its requested strategy, complementing benchmarks that test code correctness or plausibility. Related work spans code benchmarks, real-strategy generation, code hallucinations, and finance forecasting.
6 Related Work
- Functional code benchmarks use unit or repository tests [1] [4] [8] [9] [17], while LiveCodeBench addresses contamination with post-cutoff problems [5].MintEval instead generates tasks combinatorially and tests behavior over thousands of stateful steps against a reference.Source: §6 · p. 4
- QuantCode-Bench [7] uses 400 real strategy descriptions and an LLM judge for semantic alignment, reaching 70–76% single-turn and 95–98% with iterative repair.Its real tasks and broad model coverage are stronger than MintEval's; the benchmarks ask complementary questions about plausible generation and reference-based behavior.Source: §6 · p. 4
- Code-hallucination studies catalogue incorrect but plausible code [13] [16], whereas MintEval quantifies one consequential form of it.Prior finance LLM work has focused mainly on forecasting and domain language models [10] [14] [15].Source: §6 · p. 4
- MintEval's metric borrows implementation-shortfall logic by measuring the gap between intended and executed strategies with alpha differenced away.Backtest overfitting and operational failures motivate treating implementation fidelity as a first-class risk.Source: §6 · p. 4
7 Limitations
MintEval is a controlled diagnostic benchmark, not a sample of naturally occurring strategies. Its initial results are bounded by a small, single-market evaluation and LLM-written instructions, among other stated scope limits.
7 Limitations
- MintEval's controlled generation permits independent variation of state span, description length, and block type, but does not sample the distribution of real strategies.A real subset with human-written intents and human-verified reference ports is under construction to test transfer of model rankings and the state-span effect.Source: §7 · p. 4
- Version 0 uses one instrument, one data window, 800 tasks, and four models with one sample each.A human baseline and a human-rewritten instruction subset are in progress.Source: §7 · p. 4
- Instructions are LLM-written and may be friendlier to LLMs than real traders’ messages, while the slang dictionary remains under human review.The reference library, interface document, and desk conventions were written with Claude Opus 5.5's help, potentially favoring that model family.Source: §7 · p. 4
- The benchmark measures bar-level actions and complete round trips, not intra-position dynamics such as fills inside a bar beyond stop/take priority.Latency is deliberately excluded, and ambiguous instructions may still cause mismatches despite the round-trip reader.Source: §7 · p. 4
Ethics Statement
MintEval evaluates implementation fidelity rather than profitability, and its model-generated code runs in a restricted sandbox. The paper reports public market data and no personal data use.
Ethics Statement
- MintEval evaluates implementation fidelity, not profitability, and makes no investment-advice claim; most reference strategies lose money after costs.Market data are public, and no personal data are used.Source: Ethics Statement · p. 4
- Model-generated code runs in a sandbox without network or file access.The benchmark suite, prompt templates, and evaluation harness are available at the stated project repository.Source: Ethics Statement · p. 4
A 𝜏max distribution
Figure 3 shows the task set’s distribution of log10 τmax as a histogram.
A 𝜏max distribution
- Figure 3 presents the task set’s log10 τmax distribution as a histogram.Source: Figure 3 · p. 5
- The quantity shown is log10 τmax.Source: Figure 3 · p. 5
- The distribution is over the task set.Source: Figure 3 · p. 5