Source-linked AI summary
FIDES: A Concordance Protocol for LLM-Generated Trading Strategies
Arther Tian, Alex Ding, Simon Wu, Aaron Chan
TL;DR
Existing evaluations rarely check whether an LLM trading strategy’s rationale, code, and realized track record are the same object. FIDES reconciles these views through dual delivery, three concordance gaps, and a lag-one out-of-sample backtest. Across 40 strategies, concordance did not predict profit, self-assessed edges were poorly calibrated, and judge scores were not robust to swapping models.
Problem
Existing financial and code evaluations do not jointly test whether a generated trading strategy’s prose, implementation, and realized outcome agree.
Method
FIDES uses dual delivery, three concordance gaps, and a lag-one out-of-sample backtest to reconcile strategy prose, code, and track record.
Results
Across 40 strategies, concordance did not predict profit, self-reported edge was badly calibrated, and swapping the language–code judge changed say→do on more than half of items.
Takeaways & Limitations
Concordance, competence, and judge agreement should be treated as separate measurements rather than collapsed into a single performance assessment.
Takeaways & Limitations
External validity is limited by model coverage, while the runtime future-information probe fired on neither clean nor injected code and was acknowledged as a probe limitation.
Abstract
from arXiv · showhide
An LLM asked for a trading strategy returns three artifacts at once: a natural-language rationale, an executable implementation, and once run, a track record. Whether these are the same object is rarely checked. We present FIDES, a measurement protocol that treats them as three views to be reconciled rather than one deliverable to be graded. Through dual delivery, a single model call returns both a natural-language strategy with an explicit claimed edge and a self-contained strategy(df) function. FIDES executes the code in a sandbox against a lag-one out-of-sample backtest and scores three concordance gaps: say to do, do to real, and say to result. On 8 liquid US ETFs across four models plus a two-stage elicitation arm, 40 strategies, 2023 to 2024 out-of-sample, three findings stand out. First, concordance does not predict profit: only 2 of 40 strategies beat buy-and-hold, and a plain sma(50,200) rule outperforms every model's mean Sharpe. Second, self-assessment is badly calibrated: 32 of 40 strategies claim to beat buy-and-hold and exactly one does. Third, swapping the language-code judge for a second model flips say to do on more than half of items. Injecting Close.shift(-1) drops do to real by 0.33 on average, while our runtime future-information probe fired on neither clean nor injected code. We frame FIDES as a protocol for measurement fidelity, not a claim about market performance.
1 Introduction
FIDES evaluates whether an LLM trading strategy’s prose, code, and realized track record describe the same object. It combines dual delivery, three concordance gaps, and a lag-one out-of-sample backtest, while treating profit as a side metric.
- Protocol: FIDES reconciles an LLM strategy’s stated logic, generated code, and realized track record instead of grading only one artifact.The protocol uses dual delivery so prose and code originate from one model call.
- Protocol: Three gaps—say→do, do→real, and say→result—measure agreement between prose, implementation, execution, and claimed edge.Each gap is scored from 0 to 1, with 1 indicating full concordance.
- Protocol: A lag-one out-of-sample backtest anchors realized outcomes in an evaluation the model cannot game.Positions act on the prior bar, preventing trades from using information they should not have had.
- Findings: The study reports 2 of 40 strategies beating buy-and-hold, while a plain sma(50,200) rule outperformed every model’s mean Sharpe.It also reports that 32 of 40 strategies claimed to beat buy-and-hold, but exactly one did.
- Scope: FIDES frames its contribution as measurement fidelity, with external validity and model coverage explicitly delimited as future work.The protocol includes a zero-LLM rule baseline, a two-stage control, and disclosed diagnostics.
2 Related Work
Related work evaluates generated code, financial LLM capabilities, stated reasoning, and LLM judges, but does not jointly reconcile trading strategy prose, code, and outcome. FIDES adapts execute-to-verify and judge-auditing ideas to a strategy artifact whose behavior can be checked by a backtest.
- LLMs for finance and trading: Financial LLM benchmarks assess domain knowledge or pursue trading profit, rather than testing whether one strategy’s prose, code, and outcome agree.FIDES treats this concordance question as prior to and orthogonal to profitability.
- Evaluating generated code: Generated-code benchmarks use held-out unit tests, but trading strategies lack reference implementations for checking whether code matches stated rules.FIDES therefore combines execution with explicit language–code comparison.
- Faithfulness of stated reasoning: Faithfulness research shows stated rationales may not reflect the computation producing an answer, motivating FIDES’s say→do gap.FIDES moves the faithfulness question from labels to executable code whose behavior is tested by backtest.
- LLM-as-judge and its biases: Because LLM judges exhibit documented biases, FIDES audits say→do by fixing a non-candidate judge and rescoring with a second model.The disagreement is reported rather than treating either judge as an oracle.
- Backtest hazards: Finance’s overfitting, data-snooping, and look-ahead hazards motivate FIDES’s strict lag-one out-of-sample backtest and refusal to optimize profit inside the loop.The design treats these hazards as evaluation constraints.
- Trustworthy, cost-aware evaluation: FIDES extends the authors’ cheap-first, self-auditing evaluation stance to generated trading strategies through three gaps anchored by a non-gameable backtest.The object and mechanism are new within that broader evaluation approach.
3 Protocol Design
FIDES obtains prose, code, and a claimed edge in one model call, then executes the code with a lag-one out-of-sample engine and evaluates three concordance gaps. Its protocol combines language–code judging, execution checks, outcome validation, sandboxing, and controls without repairing strategies to improve performance.
- Dual delivery: A single model call produces a natural-language strategy, a self-contained strategy(df) function, and a one-sentence claimed edge.The unified elicitation avoids a separate prose-to-code translation step.
- Three concordance gaps: The protocol scores say→do, do→real, and say→result gaps, each intended to quantify agreement between successive strategy artifacts.The gaps compare language with code, code with execution, and claims with realized OOS outcomes.
- Three concordance gaps: The say→do score averages four binary checks covering direction rules, named indicators and lookbacks, exits and risk rules, and extra code rules.The fixed judge runs at temperature 0, and a literal lookback-number fraction provides an additional rule-based ceiling or diagnostic.
- Execution and leakage: The do→real score credits valid execution, nonconstant signals, and absence of static look-ahead patterns in sandboxed code.Failed runs are recorded with ρ=0 rather than crashing the evaluation, while signals are coerced, clipped, and reindexed.
- Outcome validation: The say→result score checks claimed edges against OOS Sharpe and return, requiring all triggered assertions to pass and assigning 0.5 when no testable claim is detected.For example, beating buy-and-hold requires both S > Sbh and R > Rbh.
- Non-gameable backtest: The lag-one engine trades at t using the signal from t−1, with zero-cost primary evaluation, a 10 bp turnover ablation, and buy-and-hold and sma(50,200) baselines.Strategies are neither repaired nor searched to raise Sharpe, preserving the protocol’s identification purpose.
4 Experimental Setup
The experiments evaluate generated strategies on eight liquid US ETFs using four dual-delivery models plus a two-stage elicitation arm. The setup discloses model-coverage and judge-circularity constraints while reporting concordance, OOS performance, benchmark comparisons, position density, and calibration.
- Universe and windows: The universe contains eight liquid US ETFs spanning equities, size, international markets, bonds, gold, and sectors.The tickers are SPY, QQQ, IWM, EFA, TLT, GLD, XLF, and XLE.
- Universe and windows: The primary OOS window is 2023–2024, with 2018–2022 shown only as model context and 2020–2021 used for regime robustness.The data are daily OHLCV observations on real tickers.
- Models and coverage: Four models generate 32 dual-delivery strategies, and a two-stage gpt-4o-mini arm adds 8 strategies, totaling 40 routing units.Runs use seed 42 and temperature 0.2.
- Models and coverage: Model coverage is OpenAI-heavy because Claude, Gemini, DeepSeek, Llama-70B, and Qwen-72B were unavailable in the inference catalog.The authors treat this as an infrastructure constraint and do not present the roster as a broad model comparison.
- Judge and swap: The primary say→do judge is gpt-4o-mini, while every item is also rescored by qwen-2.5-7b-instruct despite circularity on the qwen rows.Neither judge is treated as ground truth.
- Controls and metrics: Reported metrics include the three gaps, OOS Sharpe and return versus buy-and-hold, beat-buy-and-hold rate, position density, and self-assessment calibration.Calibration compares claimed versus actual beat-buy-and-hold performance.
5 Results
Across the 40-strategy evaluation, concordance and execution measures did not establish profitable trading edge, while self-assessment and judge scores proved poorly calibrated or unstable. Perturbation tests supported the static leakage diagnostic but exposed a limitation in the runtime probe.
- 5.1 Leaderboard: say→do ranged from 0.81–1.00 while do→real ranged from 0.75–1.00 across models, but say→result collapsed to 0.00 for three of five arms.The results distinguish relatively consistent language–code alignment and mostly clean execution from failure of stated edges to survive realized outcomes.
- 5.2 Concordance does not predict alpha: Only 2 of 40 strategies beat buy-and-hold on both Sharpe and return, and a plain sma(50,200) rule averaged 0.583 OOS Sharpe, above every model mean.Strategies with say→do=1.0 beat buy-and-hold once in 22 (4.5%), versus once in 18 (5.6%) for say→do<1.0.
- 5.3 Self-assessment is badly calibrated: 32 of 40 strategies claimed to beat buy-and-hold, but exactly 1 did, a 3.1% hit rate.The lone two-stage say→result score of 0.50 mostly reflected the “no testable claim detected” default rather than a confirmed edge.
- 5.4 The language–code judge is not an oracle: Under the original judge, say→do spanned 0.81–1.00; under Qwen-7B, it compressed to 0.72–0.75, with exact agreement on 16 of 40 items.The mean absolute difference was 0.15, and model rankings collapsed when the judge was swapped.
- 5.5 Look-ahead dose–response, and an honest negative: Injecting Close.shift(-1) reduced mean do→real by exactly 0.333 across 35 running strategies, and the static look-ahead check caught the injected leakage in every case.The injection produced the intended dose–response for the execution-anchored diagnostic.
- 5.5 Look-ahead dose–response, and an honest negative: The independent runtime probe fired on neither clean nor injected code, 0 of 40 both times, because its shuffle boundary and observation window did not overlap.The paper treats this as a limitation of the probe rather than evidence of clean execution and retains the static check.
6 Discussion
FIDES separates concordance from competence: a strategy can faithfully implement its stated logic yet fail to beat buy-and-hold. The authors recommend reading the three gaps separately, while limiting claims by the study’s narrow sample and unresolved judge and parser issues.
- Concordance and competence are different axes: faithful code can still produce a strategy that does not beat holding the asset.The authors identify conflating internal consistency with performance as a deployment risk.
- Practitioner guidance: do→real is a hard execution gate, while say→result must be checked against a non-gameable backtest rather than accepted at face value.
- Practitioner guidance: say→do should be reported with both the fixed judge’s score and a swapped-judge number because a single judge is only one measurement.
- Limitations: The study covers 40 strategies, eight liquid US ETFs, daily rule-based strategies, and limited model coverage, so it reports effects rather than confidence intervals.
7 Conclusion and Future Work
FIDES reconciles stated logic, generated code, and realized track records through three concordance gaps anchored by a backtest. The study finds weak links between concordance and profit, poor calibration of self-reported edge, and unresolved external validity, motivating broader and stronger future tests.
- FIDES treats an LLM trading strategy as three artifacts—stated logic, generated code, and realized track record—and scores three concordance gaps.The gaps are anchored by a backtest the model cannot game.
- Across 8 ETFs and four models, concordance did not predict profit, self-reported edge was badly calibrated, and the LLM-scored gap was not robust to swapping judges.The study reports 1 of 32 self-reported claims held.
- A look-ahead dose–response validated the execution gap, while the runtime probe did not work and was disclosed as a limitation.
- Future Work: Future work should broaden model coverage, add seeds, replace the keyword parser, repair or replace the runtime probe, and extend evaluation toward portfolios and genuine forward tests.
A Configuration
The supplied configuration passage identifies Table 7 as the frozen v0 configuration.
- Table 7 is labeled “Frozen v0 configuration.”
B The two “winners”
Only two of forty strategies beat buy-and-hold, and both were on TLT, but they represent different interpretations of a raw win count. One was a sparse mean-reversion rule with positive out-of-sample performance; the other stayed flat while TLT fell.
- Only 2 of 40 strategies beat buy-and-hold, and both winners were on TLT.
- Real-ish: The gpt-4o TLT winner used RSI mean reversion, a 20-day moving-average filter, and a volume liquidity gate.
- Real-ish: 0.759 OOS Sharpe versus buy-and-hold −0.091, with returns of +0.084 versus −0.055, characterized the gpt-4o TLT winner.Its position density was 0.098.
- Degenerate: The gpt-4o-mini two-stage TLT winner stayed flat, producing zero position and zero return while correctly making no claim to beat buy-and-hold.
- Degenerate: The flat strategy beat buy-and-hold only because TLT fell and holding cash did not, making the calibration count 1 rather than 2.