PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark

Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, Arsenii Bobovnikov

arXiv:2609.28090v1cs.CLcs.AIcs.CEcs.SE

TL;DR

Backtest auditing must balance flaw detection with letting matched clean strategies pass, since false accusations add review burden. The paper evaluates this with a paired benchmark and deterministic scoring across audit dimensions. DeepSeek's clean-aware prompt removes clean-control false positives without reducing code recall, while the budget anchor still over-flags clean controls.

  • Problem

    Backtest auditing must detect methodological flaws while letting clean strategies pass, because false positives add review threads and can obscure the backtest needing repair.

    Source: §1 · p. 1

  • Method

    The paper builds a 96-item paired benchmark with matched clean controls and uses a deterministic validator to score recall, false positives, evidence, fixes, specificity, and utility.

    Source: §2 · pp. 1–2

  • Results

    DeepSeek clean-aware code flags 0/48 clean controls versus 10/48 under closed prompting, while both prompts retain 48/48 code recall.

    Source: §3 · p. 2

  • Takeaways & Limitations

    Backtest-auditor benchmarks should report flaw recall, clean false positives, evidence, fixes, and confidence separately rather than relying on a single aggregate.

    Source: §4 · p. 4

    The benchmark uses synthetic paired edits over fixed strategy skeletons, not externally collected backtest notebooks, and does not certify real trading strategies.

    Source: Limitations · p. 4, Ethics Statement · p. 4

Abstract

from arXiv · show

Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.

1 Calibration as Much as Flaw-Finding

Backtest review requires calibration as well as flaw detection: an auditor must identify methodological errors while letting clean strategies pass. The paper studies this with matched controls, distinguishing its evaluation from prior work on backtest and LLM review.

1 Calibration as Much as Flaw-Finding

  • A useful LLM backtest auditor must detect methodological flaws while letting clean strategies pass, because false positives add review threads and can obscure the backtest needing repair.Closed-set recall alone is misleading for this calibration problem.Source: §1 · p. 1
  • The benchmark pairs each flawed item with a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail.It isolates a review skill and does not certify trading systems.Source: §1 · p. 1
  • Unlike prior quantitative-finance and LLM-review benchmarks, this work jointly evaluates matched clean controls, prose and code surfaces, canonical flaw labels, and deterministic evidence and fix scoring.Prior work covers backtest overfitting, data snooping, survivorship bias, backtest generation, look-ahead mitigation, issue-level critique, and LLM-assisted review (White, 2000; Harvey et al., 2016; Bailey et al., 2017; Lopez de Prado, 2018; Wang et al., 2026; Li et al., 2026; Chaudhuri et al., 2026; Ho et al., 2026; Zurawicki et al., 2026).Source: §1 · p. 1

2 Benchmark and Validator

The benchmark pairs flawed backtests with clean controls containing protective methodology constructs, then evaluates audits across prompt conditions and prose and code surfaces. A deterministic validator separately scores false positives, evidence, fixes, all-three specificity, and utility.

2 Benchmark and Validator

  • The 96-item benchmark contains 48 clean/flawed pairs, with six flawed examples per class, three difficulty strata, and prose and code surfaces across eight canonical flaws plus a clean label.Clean controls include protective constructs such as purged and embargoed splits, explicit one-way costs, half-spread slippage, and non-overlapping labels.Source: §2 · p. 1
  • The prompt grid crosses open, closed, and clean-aware conditions with prose and code; the primary auditor is DeepSeek V4 Flash at temperature 0.The code-slice ladder adds gpt-4o-mini, gemini-2.5-flash-lite, and gpt-4.1-mini; all 1440 calls parse after a token-cap rerun.Source: §2 · p. 1
  • The paired design holds the strategy and reporting scaffold fixed while contrasting legitimate split, lag, and cost lines with a planted look-ahead edit.The figure depicts an open code audit flagging the flawed edit while falsely accusing its matched clean control.Source: Figure 1 · p. 2
  • Table 1 is the primary prompt-by-surface confusion matrix, with flawed cells showing true positives and false negatives and clean cells showing false positives and true negatives.Threads are expected unnecessary reviews per 100 clean submissions.Source: Table 1 · p. 2
  • The deterministic validator separately scores clean-control false positives, evidence hits on planted target lines, class-specific fix credit, all-three specificity, and utility.It defines review burden as 100 times the clean-control false-positive rate and reports 95% Wilson score intervals.Source: §2 · p. 2

3 Results: Recall Is Easy, Calibration Is Not

Clean-aware prompting preserves DeepSeek’s perfect code recall while eliminating clean-control false positives, but fix relevance and model choice still separate audit quality. Open code prompts produce many unsupported flags and miss flaws concentrated in particular classes.

Open prompts

  • Open code audits detect 35/48 flawed items while creating 45 unnecessary review threads per 48 clean controls, or 93.8 per 100 clean submissions.The primary auditor misses 13 flawed code items while raising 45 unsupported flags, so the flagged pile contains more clean strategies than rigged ones.Source: §3 · pp. 2–3

Prompt calibration

  • 100.0% code recall is unchanged as DeepSeek’s clean-aware prompt reduces code FPR from 20.8% to 0.0% and prose FPR from 2.1% to 0.0%.Both closed and clean-aware prompts reach 48/48 code recall; the closed prompt flags 10/48 clean controls, versus 0/48 for clean-aware.Source: §3 · p. 2

Audit quality

  • 87.5% clean-aware code all-three specificity reflects 100.0% evidence localization but only 87.5% fix relevance.The model localizes every flagged item correctly yet proposes an off-class remedy in six; prose reaches 100.0% all-three specificity.Source: §3 · p. 2

Model calibration

  • Three of four endpoints tie at 100.0% recall, yet their clean-control rates span 79 points.DeepSeek and Gemini flag 0/48 clean controls, GPT-4.1 mini flags 8/48, and GPT-4o mini flags 38/48; Table 2 reports the clean-aware code slice.Source: §3 · pp. 2–3, Table 2 · p. 3

Open prompts: flaw classes

  • Open code recall is 1/6 for full-sample preprocessing and 2/6 for regime overfit, despite 6/6 recovery for look-ahead, survivorship, and data snooping.Difficulty does not explain the misses: open code recall is 12/16 for obvious, 11/16 for moderate, and 12/16 for subtle items.Source: §3 · p. 3

Reviewer persona

  • Across four reviewer roles, recall ranges from 95.8% to 100.0% while clean-control flags range from 0/48 to 6/48.The two adversarially connoted roles account for 11 of 12 false positives; risk_manager alone misses a flawed item.Source: Table 3 · p. 3, §3 · p. 3

Confidence and calibration

  • Mean self-reported confidence tracks false-positive rate in the wrong direction: roles flagging more clean controls report lower confidence (0.911 and 0.928 versus 0.953 and 0.952), so confidence filtering discards better-calibrated roles’ flags.Source: §3 · p. 3

Surface comparison

  • Under open prompts, prose recall exceeds code recall (81.2% against 72.9%) and prose FPR is less than half the code rate (37.5% against 93.8%).With closed prompts, the recall gap closes to zero while the FPR gap persists: 2.1% for prose against 20.8% for code.Source: §3 · p. 3

4 Discussion

Under deterministic paired edits, prompt and model choice separate high flaw recall from clean-control calibration; these results do not establish production validation. Backtest-auditor benchmarks should report recall, clean false positives, evidence, fix, and confidence separately.

4 Discussion

  • Benchmarks should report flaw recall, clean false positives, evidence, fix, and confidence separately because one aggregate would rate all four endpoints as near-solved.The recommendation is limited to backtest-auditor benchmark design.Source: §4 · p. 4
  • Prompt and model choice separate high flaw recall from clean-control calibration under deterministic paired edits.Source: §4 · p. 4
  • The benchmark does not show that an LLM can validate a trading system in production.Source: §4 · p. 4

Limitations

The study uses a small, synthetic set of paired edits over fixed strategy skeletons and single-call audits, with no completed human reference rate. Its lexical fix scorer and saturated recall results also constrain interpretation.

Limitations

  • Only six flawed examples per class make per-class rates move in 16.7-point steps and leave Table 2 Wilson intervals wide.Source: Limitations · p. 4
  • Synthetic paired edits over fixed strategy skeletons are not externally collected backtest notebooks.Source: Limitations · p. 4
  • Each audit is one temperature-0 call rather than a multi-turn review, and the prepared 16-item human baseline was not completed.Source: Limitations · p. 4
  • The strict lexical scorer may under-credit unusual fixes; alias-only matches make up 56 correct responses, or 4.6%, leaving the synonym table a live sensitivity.Source: Limitations · p. 4
  • Many clean-aware results saturate at 100% recall, while GPT-4o mini still over-flags clean code controls.Source: Limitations · p. 4

Ethics Statement

The benchmark evaluates recognition of planted methodology flaws in synthetic backtests under matched controls, not trading-system validity. It redistributes no market data, broker records, or proprietary strategies and makes no profitability claim or investment recommendation.

Ethics Statement

  • All benchmark backtests are synthetic, with no market data, broker record, or proprietary strategy redistributed.Source: Ethics Statement · p. 4
  • The benchmark measures recognition of planted methodology flaws under matched controls; it does not certify trading systems or establish that a real strategy is valid.Source: Ethics Statement · p. 4
  • The paper makes no claim about profitability and offers no investment advice.Source: Ethics Statement · p. 4

A Per-Class Recall Without the Taxonomy

The closed prompt recovers every flaw class on both surfaces, whereas open-prompt recall falls short, particularly for full-sample preprocessing and regime overfit.

A Per-Class Recall Without the Taxonomy

B Recall Against False Positives by Surface

Figure 3 compares primary-auditor recall with clean-control false positives across prompt modes and surfaces. The clean-aware prompt gains recall over the open prompt while also reducing false positives.

B Recall Against False Positives by Surface

Loading 2609.28090v1…