PaperlayerView exact revision on arXiv

Source-linked AI summary

JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making

Zhongwei Xie, Xingsu Li, Jingqiao Yu, Zihao Wang, Hong ChiTat, Zhaowei Wang, Weizhi Du, Yuxuan Liu, Huihao Jing, Jiaxin Bai, Yangqiu Song

arXiv:2610.19589v1cs.AI

TL;DR

Repeated event forecasting requires extracting predictive signals from noisy evidence and determining whether forecasts yield value across successive decisions. JockeyBench evaluates LLM agents in historical Japanese horse racing using time-gated evidence, priced tools, paired controls, and two forecasting workflows. Revision gains occur in some configurations, but every configuration loses on average under capped-Kelly settlement.

  • Problem

    The paper asks when predictive advantage translates into value across repeated decisions and how revision and information costs affect that value.

    Source: §1 · p. 1

  • Method

    JockeyBench combines time-gated Japanese racing evidence and priced tools with predictor and evidence-acquisition workflows, paired frozen controls, and separate forecast, utility, and betting measures.

    Source: §1 · pp. 1–2

  • Results

    Seven configurations had significant revision gains and two had declines after correction, while every configuration had negative mean capped-Kelly profit.

    Source: §4.1.1 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The findings support evaluating predictive competence, adaptation, resource use, and decision value separately in repeated-event agent evaluations.

    Source: §5 · p. 8

    The evaluation focuses on structured horse-racing forecasts, and generalization to other sports, unstructured information, and broader decision tasks remains to be established.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Repeated event forecasting requires extracting predictive signals from noisy evidence and evaluating their value across successive decisions. We introduce JockeyBench, a benchmark and evaluation protocol that studies this problem through historical Japanese horse racing. The protocol combines time-gated evidence, priced tools, paired frozen-program controls, and two workflows: executing an LLM-written predictor or acquiring evidence for a fresh per-race LLM forecast. Evaluation separates forecast quality, benchmark-specific net utility, and retrospective betting returns and risk under fixed decision rules. Across 28 model--scaffold configurations on 324 races, paired tests identify seven revision gains and two declines after multiple-testing correction. Yet every configuration has negative mean profit under capped-Kelly settlement. In a four-model comparison, the reusable-program workflow has higher mean betting profit and lower research fees, although both workflows lose on average. Further diagnostics reveal that mean probability bias reverses on selected bets and that profitable individual runs can depend on concentrated returns. These findings motivate evaluating predictive competence, adaptation, resource use, and decision value separately when assessing agents for repeated event forecasting.

1 Introduction

JockeyBench studies whether LLM agents can extract useful signals from noisy historical evidence and translate forecasts into value across repeated decisions. It pairs time-gated racing data and priced evidence access with distinct forecasting workflows, controls, and measures of prediction, cost, and betting outcomes.

1 Introduction

  • JockeyBench uses linked Japanese racing records, historical information cut-offs, delayed outcome release, and excludes odds and expert ratings from agent inputs.The setting permits retrospective evaluation using official dividends under fixed betting rules.Source: §1 · p. 1
  • The protocol compares executable LLM-written predictors with evidence-acquisition programs followed by fresh per-race LLM forecasts, and pairs revisions with frozen initial programs.The paired controls distinguish revision gains from final forecast quality.Source: §1 · p. 1
  • Evaluation separates predictive quality, benchmark-specific net utility, resource fees, and betting profit and risk rather than treating them as interchangeable outcomes.Net utility combines predictive reward and research fees under a common tariff; established forecast and risk measures retain separate roles.Source: §1 · p. 2
  • Across 28 configurations on 324 races, seven revision gains and two declines were significant after correction, yet every configuration had negative mean capped-Kelly profit.Program had higher mean betting profit and lower research fees than Adaptive Acquisition in a four-model comparison, although both workflows lost on average.Source: §1 · p. 2

2 Related Work

Related evaluations address sustained decisions, program revision, evidence acquisition, forecasting, and financial outcomes, but differ in which capabilities and outcomes they assess. JockeyBench brings these strands together for repeated racing forecasts under priced evidence access.

Long-Horizon Agents and Cost-Aware Evaluation

  • JOCKEYBENCH compares daily program revision with frozen initial programs and charges for evidence, tool calls, and computation under a fixed tariff.The ten-day evaluation window captures short-term program revision.Source: §2 · pp. 2–3

Related evaluations

  • Table 1 compares reviewed protocols on design features including outcome revision, dual workflows, payoffs, and risk metrics.Its checks indicate inclusion; crosses indicate absence or that a feature is unreported.Source: Table 1 · p. 3

Forecasting and Financial Decision-Making

  • Forecasting benchmarks cover unresolved questions, live events, timestamped markets, document and clinical evidence acquisition, and sequential sports betting.JOCKEYBENCH compares LLM-written predictors with evidence collectors followed by fresh per-race forecasts, excluding odds and expert ratings from agent inputs.Source: §2 · p. 3

Horse-Race Modeling and Statistical Inference

  • Horse-racing research includes conditional-logit winner models, probability models using win-pool information, and models combining fundamental and public betting probabilities.Proper scoring rules, Kelly-based analyses, and cluster-robust inference inform forecast evaluation, settlement, and uncertainty estimation.Source: §2 · p. 3

3 The JockeyBench Protocol

JockeyBench combines time-gated historical racing data with persistent forecasting and acquisition workflows, frozen controls, priced tools, and separate measures of prediction, resource use, and betting outcomes.

3.1 Dataset and information controls

  • The database links twelve years of Japanese racing records, assigns records their public-availability times, and exposes 97 whitelisted objective fields through priced tools.Odds, dividends, expert ratings, free text, external tools, and program network access are excluded.Source: §3.1 · p. 3

3.2 Program workflow and frozen controls

  • Each day, the saved program forecasts races before outcome release, then receives released results and may be revised for the next day.Each revised trajectory is paired with its initial program run unchanged on the same schedule and tariff; failures fall back to a previous valid program or uniform forecasts.Source: §3.2 · p. 4

3.3 Adaptive Acquisition workflow

  • Adaptive Acquisition retains an evidence-selection program that chooses information for each race, then passes it to a fresh LLM session to produce sealed forecasts.The acquisition program cannot inspect raw evidence values or construct a predictor; released aggregate feedback may inform later program updates.Source: §3.3 · p. 4

3.4 Evaluation metrics

  • Forecast evaluation includes fallbacks and reports logarithmic scoring relative to uniform, top-1, and top-3 precision alongside resource use.For Program, benchmark-specific net utility combines predictive reward with a weighted research fee based on billable evidence cells, tool calls, and CPU time; provider charges are separate.Source: §3.4 · pp. 4–5
  • Retrospective flat-bet and capped-Kelly outcomes use odds withheld from agents, while drawdown and profit-based Sharpe summarize the mean profit-and-loss path.The workflow comparison uses common scoring and accounting rules; the authors caution that Net is synthetic utility, not a strictly proper scoring rule.Source: §3.4 · p. 5

4 Experiments and Analysis

Across 28 Program configurations and four workflow-comparison models evaluated on 324 races, the experiments distinguish forecast quality, revision, resource costs, and decision outcomes. Program has higher mean betting P&L than Adaptive Acquisition, but every configuration loses on average under capped-Kelly settlement.

4.1.1 Program forecasts

  • Seven revision gains and two declines in mean log gain remain significant after Holm correction across 28 tests.The largest gain, +0.359 for Nemotron-3-Ultra-550B, leaves its revised mean below uniform at −0.083; the two highest-scoring configurations show no revision gain.Source: Table 2 · p. 6, §4.1.1 · p. 6
  • 0.525 nats is GPT-6.1-Sol’s mean log gain, exceeding the card-and-history reference R2 at 0.343.Agents produce 40,428 predictor outputs among 44,712 forecasts, while some delivered programs score below uniform and two models never deliver a predictor.Source: §4.1.1 · pp. 5–6, Table 2 · p. 6
  • 0.608 nats for broad factual features exceeds the agent-inspired ensemble’s 0.523, while broad factual features remain the strongest offline reference across all three periods.The closing market has the highest log gain in each period; the expanded R3 features underperform R2 in every period.Source: §4.1.1 · p. 6, Table 3 · p. 7

4.1.2 Adaptive Acquisition forecasts

  • Adaptive Acquisition does not consistently improve forecasts: DeepSeek’s observed log gain is 0.256 versus Program’s 0.232, while Program leads all three metrics for GLM, GPT-6-Astra, and MiMo.The workflow comparisons are descriptive; Adaptive cohorts contain six, one, two, and one completed trajectories, respectively, and share historical outcomes.Source: §4.1.2 · p. 7

4.2 Decision-making performance

  • Every configuration loses on average under capped-Kelly settlement; across four models, Program has higher mean betting P&L than Adaptive Acquisition, though both workflows lose.Program also has lower mean research fees across the four models.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
  • Kelly-selected bets reverse the overall probability-bias pattern: DeepSeek’s predicted and observed win rates are 18.14% and 21.25% overall, versus 17.83% and 7.82% on selected bets.These pooled mean gaps are descriptive, not a full calibration test or a causal explanation of losses.Source: §4.2 · p. 8
  • DeepSeek Adaptive R1 earns JPY 1,470 under Kelly, while the other five completed trajectories lose JPY 47,568 on average; three winning tickets account for 55.35% of R1’s gross returns.MiMo’s Kelly profits range from JPY −70,500 to JPY +73,080 across five trajectories; these repeated executions use fixed outcomes, not independent temporal replications.Source: §4.2 · p. 8
  • Research fees reverse DeepSeek’s gross-reward comparison: Adaptive Acquisition gains approximately 1,000 score-yen but incurs 4,551 more in fees per trajectory.For GLM, GPT-6-Astra, and MiMo, Adaptive Acquisition has both lower gross reward and higher fees; under the common rescoring rule, Program has higher mean net scores across all four models.Source: §4.2 · p. 8

5 Conclusion

JOCKEYBENCH combines a historical racing resource with a protocol for evaluating persistent forecasting programs, evidence acquisition, and revision under constraints. The evaluated agents recover useful signals, but no configuration is profitable on average under capped Kelly, and generality beyond the shared historical window remains untested.

5 Conclusion

  • JOCKEYBENCH combines historical racing data with evaluation of persistent forecasting programs, evidence acquisition, and revision under information and resource constraints.Its empirical contribution distinguishes execution from predictive competence, revision gains from final quality, predictive reward from net utility, and overall probability bias from errors on selected bets.Source: §5 · p. 8
  • Agents recover useful signals, yet no configuration is profitable on average under capped Kelly.The findings support paired frozen controls, explicit cost decomposition, and repeated decision-level diagnostics in agent evaluation.Source: §5 · p. 8
  • Generality beyond the shared historical window remains to be tested.Source: §5 · p. 9

Limitations

The evaluation is limited to structured horse-racing forecasts and the specific model–scaffold configurations tested. Performance under other environments, pricing schemes, model versions, or deployment routing is not established.

Limitations

  • Generalization beyond structured horse-racing forecasting—to other sports, unstructured information, and broader decision tasks—remains unestablished.The reported results also depend on the evaluated model–scaffold configurations; other interfaces, reasoning budgets, or model versions may change relative performance.Source: Limitations · p. 9
  • Agent adaptation of information acquisition and resource allocation under alternative pricing schemes remains unexplored.Research costs are compared under a common tariff, while fixed-history evaluation does not assess difficulty-based model routing in deployment.Source: Limitations · p. 9

Ethics Statement

The study evaluates forecasting through historical simulation, with no real bets placed, and notes ethical, data-access, and release constraints. The appendix guide identifies where benchmark details, extended results, prompt materials, strategy case studies, and reproducibility materials appear.

Ethics Statement

  • No real bets were placed; betting outcomes were calculated retrospectively from historical results and official dividends.Score-yen is a synthetic evaluation unit, and the source archive was accessed through a commercial membership.Source: Ethics Statement · p. 9
  • Raw database files and private identifier mappings remain on the evaluation host, while pseudonymized tool responses use configured model-provider routes.External processing and redistribution remain subject to permission verification.Source: Ethics Statement · p. 9

Guide to the appendix

  • The appendix guide directs readers to benchmark details and extended results, followed by prompts, strategy case studies, and reproducibility materials.It lists these respectively in Appendices A, B, C, D, and E.Source: Guide to the appendix · p. 12

A.1 Data, preprocessing, and information timing

The appendix describes data provenance and agent-facing materials, then specifies the Program and Adaptive Acquisition workflows, their execution constraints, and validation procedures. Both workflows follow the same information-release schedule.

A.1 Data, preprocessing, and information timing

  • The archive contains 7.5 million cleaned records traced to hashed source files, with field parsing, historical links, and pseudonymization documented.Files are sliced by byte position before CP932 decoding; jockey and trainer links follow race declarations rather than later master snapshots.Source: §A.1 · p. 12

A.2 Agent workflows and execution

  • Both workflows use the same information-release schedule; Program executes a saved predictor that the LLM may revise after each released day.The initial predictor is constructed by an LLM, and race forecasts are produced by the saved program.Source: §A.2 · p. 12, §A.2.1 · p. 12

A.2.1 Program

  • Program candidates must return complete rankings and win probabilities, pass interface and execution checks, and invalid candidates retain the previous valid version.Nondeterministic or exactly uniform outputs prompt repair but do not alone determine rejection or selection by accuracy; failures fall back to an earlier valid program or uniform output. Persistent source and parameters are retained under sandbox restrictions and phase-specific allowances.Source: §A.2.1 · p. 12

A.2.2 Adaptive Acquisition

  • Adaptive Acquisition programs collect broker-managed evidence through bounded SDK operations, while raw observations remain with the broker and unreleased outcomes stay withheld.The program can inspect query counts, costs, coverage, missingness, freshness, and available history, but returns evidence rather than rankings or probabilities.Source: §A.2.2 · pp. 12–13

A.1 Data, preprocessing, and information timing

  • Figure 4 illustrates synthetic preprocessing: identifiers are replaced, dates made relative, values typed, and missing-value reasons made explicit; excluded fields and withheld results are shown.Table 4 describes fields available to agents and programs, including pseudonymous IDs, categorical codes without a codebook, and nulls with reasons.Source: Figure 4 · p. 13, Table 4 · p. 13

A.2.2 Adaptive Acquisition

  • A fresh LLM session forecasts from the basic card and acquired evidence; after outcomes release, an updater may validate and save a candidate or retain the current program.Validation uses synthetic cards rather than real-race outcomes or accuracy-based selection; invalid candidates leave the previous valid program unchanged.Source: §A.2.2 · pp. 13–14

A.3 Models, configurations, and evaluation cohorts

This section documents the model cohorts, reference predictors, scoring and settlement procedures, comparison criteria, and forecast-quality diagnostics, and reports forecast-delivery records and paired revision summaries for Adaptive Acquisition and Program.

A.3 Models, configurations, and evaluation cohorts

  • The main cohort requires complete 324-race daily-revision and frozen-program runs across 28 configurations, with matched initial-program fingerprints and recorded run provenance.Workflow analyses use 31 trajectories across four models; DeepSeek Adaptive R1–R7 completed six of seven attempts, while the 24-race R5 failure remains in failure accounting without imputed full-window results.Source: §A.3 · p. 14, Table 6 · p. 16

A.4 Forecasting references

  • Offline references use hand-built, agent-inspired, and broad factual features, with time-ordered training splits and conditional-logit or boosted-model fitting.The autumn FAGENT logit did not report convergence at its 400-iteration limit; archived predictions and scores were retained. None of the feature representations includes market odds or expert forecasts.Source: §A.4 · pp. 15–16

A.5.1 Data tools and tariff

  • The tariff charges evidence cells, data-tool calls, and sandbox CPU time, while Adaptive Acquisition accounting tracks native acquisition identity and cached evidence to determine marginal charges.Common workflow rescoring applies the shared tariff to saved predictions; its gross score combines smoothed log gain and top-1 reward, while provider charges are excluded and score-yen remains separate from betting profit.Source: §A.5.1 · p. 17, §A.5.2 · p. 18

A.5.3 Flat-WIN settlement

  • Flat-WIN and capped-Kelly settlement use sealed predictions and official dividends, with capped-Kelly stakes based on the last pre-race odds snapshot and a 1% bankroll cap.Kelly sizing was selected retrospectively, and the archived odds are excluded from agent inputs; workflow Kelly comparisons use original saved probabilities and do not smooth or renormalize them.Source: §A.5.5 · p. 18, §A.5.3 · p. 18

A.5.5 Workflow comparison under shared betting rules

  • Workflow betting outcomes are simulated per trajectory, while profit and risk summaries use mean-path measures and exclude research fees and provider charges.Kelly results are descriptive archive-clock counterfactuals because quote publication times were not independently verified; incomplete trajectories receive no imputed full-window financial values, and repeated runs share historical outcomes.Source: §A.5.5 · pp. 18–19

A.5.6 Paired statistical inference

  • Paired inference computes day-level mean contrasts and applies Holm correction to two-sided t-test probabilities.The appendix also defines review criteria spanning time-gated information, predictive code, feedback and outcome revision, dual workflows, financial outcomes, risk, priced tools, and frozen controls.Source: §A.5.6 · p. 19, §A.6 · p. 19

B.1.1 Forecast quality across workflows

  • Extended diagnostics show individual-run forecast quality and net scores, but run spread reflects session variability because runs share outcomes.Additional reference-predictor evaluations cover January–June and autumn 2024 windows; Table 8 is explicitly an earlier 25-configuration export, not the full main cohort.Source: §B.1 · p. 19, Figure 5 · p. 20, Table 8 · p. 21
  • Adaptive Acquisition's direct-output rates were DeepSeek 1,918/1,944 (98.66%), GLM 314/324 (96.91%), GPT-6-Astra 644/648 (99.38%), and MiMo 322/324 (99.38%).Repaired outputs numbered 12, 7, 4, and 2, respectively; DeepSeek also had ten execution fallbacks and four explicit uniform submissions, while GLM had three explicit uniform submissions.Source: §B.1.1 · p. 22
  • Program recorded predictor outputs for DeepSeek in 1,920/1,944 cases and for each other model in 1,620/1,620 cases.DeepSeek's remaining 24 outputs used no_strategy deliveries with uniform probabilities; Program predictor categories are not equivalent to Adaptive Acquisition direct_predictor categories, and the lack of a comparable Program repair category does not establish zero repairs.Source: §B.1.1 · p. 22

B.2 Revision and frozen controls

  • Table 2 reports direct A and S means and paired revision contrasts for 138 matched pairs, while Figure 5 shows individual A trajectories and paired S means.Inference follows Appendix A.5.6, and executed strategy changes are illustrated in Appendix D.Source: §B.2 · p. 22

B.2.1 Matched Frozen acquisition controls

Matched acquisition controls show mixed net-score and forecast-quality effects despite lower research fees, while retrospective capped-Kelly returns remain negative on average. Selected-bet diagnostics further qualify how probabilities and returns should be interpreted.

B.2.1 Matched Frozen acquisition controls

  • Adaptive has lower research fees and gross reward in all three matched pairs, but net-score differences are mixed: +5,373.76, +68.11, and −203.99.These matched scores are calculated before display rounding.Source: §B.2.1 · pp. 22–23, Table 12 · p. 24
  • Forecast-quality rankings are mixed across the three matched pairs, and DeepSeek R1’s JPY 43,510 Kelly P&L improvement accompanies a JPY 63,980 drawdown increase.DeepSeek R1 and Astra R1 improve Top-1 while reducing log gain; DeepSeek R2 slightly improves log gain and Top-3 while reducing Top-1.Source: §B.2.1 · p. 23, Table 12 · p. 24
  • The three matched comparisons are descriptive, and separately generated forecasts plus few matched initial policies prevent interpreting them as a pure causal effect of daily updates.Program revision also combines outcome feedback, analysis, data access, and debugging; the workflow comparison jointly varies forecasting mechanism and information interface.Source: §B.2.1 · p. 24

B.3 Retrospective returns and risk

  • Every configuration’s mean capped-Kelly profit is negative, although 13 of 138 trajectories finish in profit.The cap binds on 96% of placed bets, and trajectories bet on 166 of 324 races on average.Source: §B.3 · p. 25

B.3.1 Calibration and bet selection

  • Adaptive Acquisition’s mean probability is below observed win frequency among all non-refunded top picks but above it among Kelly-selected bets across all four models.Program also overpredicts average wins in all four Kelly subsets; these descriptive gaps do not establish overall calibration quality or explain betting losses.Source: §B.3.1 · p. 25
  • Equal JPY 100 stakes on the same 132 non-refunded Adaptive Kelly selections yield −55.38% ROI, versus −59.00% under actual Kelly stakes.This reweighting of fixed selections is not a new simulation and does not identify a causal effect of stake weighting.Source: §B.3.1 · p. 25

B.3.2 Return concentration across DeepSeek repeats

Across DeepSeek Adaptive Acquisition repeats, a profitable run depended on a small number of winning tickets, while the other completed runs lost on average. Probability quality and realized betting returns therefore tell different parts of the story.

B.3.2 Return concentration across DeepSeek repeats

  • The diagnostic covers six completed DeepSeek Adaptive Acquisition trajectories: R1, R2, R3, R4, R6, and R7.Cohort membership and completion accounting are reported in Appendix A.3.Source: §B.3.2 · p. 25
  • JPY 1,470 in capped-Kelly profit for R1 contrasts with mean losses of JPY 47,568 across the other five completed trajectories.R1's three largest winning-ticket gross returns account for 55.35% of its total gross returns; odds of at least 10 contribute JPY 32,700 net profit in R1 versus a mean loss of JPY 32,050 in the other five.Source: §B.3.2 · p. 25
  • R1's Kelly-selected picks overpredict wins: mean raw saved probability is 17.88%, against an observed win rate of 9.03%.R2, R3, R4, and R6 have lower full-vector Brier scores than R1 but all lose under Kelly; these observations do not establish a reproducible profitable edge.Source: §B.3.2 · p. 27
  • Figure 9 summarizes ticket-return concentration, odds-band profits, trajectory profits, and selected-pick probabilities.Top-k refers to the k winning tickets with the largest gross returns, not a forecast accuracy metric; other-five odds-band bars show per-trajectory means.Source: §B.3.2 · pp. 25, 27, Figure 9 · p. 28

B.4 Research costs and synthetic utility

This section examines research costs and synthetic utility across workflows, including how fees and hit rewards can affect configuration comparisons. It also describes revision trajectories, where probability, feature, and weight changes can produce different effects on confidence, rankings, and outcomes.

B.4 Research costs and synthetic utility

  • Top-six configurations average approximately 14–17 score-yen in fees per race; fees and hit rewards can reorder net-score rankings relative to log gain.Execution time and exploration dominate fee decomposition; Claude Opus 5.5 has lower log gain than GPT-6-Sol but a higher net score.Source: §B.4 · p. 28
  • Program achieves higher mean net scores across all four models, as Adaptive Acquisition’s higher fees offset its gross-reward advantage for DeepSeek and its lower gross rewards for the other three.Table 16 reports mean gross reward, weighted research fees, and net score per completed 324-race trajectory.Source: §B.4 · p. 28, Table 16 · p. 29

C PROMPT TEMPLATES AND SUBMISSION INTERFACES

  • Reference prompts, tool schemas, and configuration are supplied in a supplement, with provenance and scope described in Appendix C.1.The model is called during initial construction and subsequent revision sessions, with at most one repair session per phase; forecast execution calls the saved program.Source: C PROMPT TEMPLATES AND SUBMISSION… · p. 30

C.1 PROGRAM PHASE PROMPTS

  • Program forecasts use inputs available by the previous evening’s 21:00 JST cutoff, while reflection runs after the day closes and before the next day’s cards exist.The prompt also defines net score as gross score minus incurred fees and specifies probability smoothing and a hit bonus for validated predictor delivery.Source: §C.1 · p. 30

C.2 TASK CONTEXT AND WORKFLOW DIFFERENCES

  • Program revision updates the saved program using released daily feedback, whereas Adaptive Acquisition updates an evidence collector and uses a separate forecasting session.The Program task object supplies the current program, notes, validation feedback, recent errors, and cost ledger; S skips revision while retaining time-varying legal inputs.Source: §C.2 · p. 34

C.3 MINIMAL SUBMISSION EXAMPLE

  • The minimal submission example returns uniform probabilities and is classified as silent_uniform, receiving no program-delivery hit bonus.Source: §C.3 · p. 34

D CASE STUDIES OF EXECUTED STRATEGIES

  • Program case studies compare daily revision with a matched frozen initial program over 324 races, while the acquisition case uses a DeepSeek Frozen-P0/Adaptive R1 control.The cases summarize executed programs, saved parameters, and query schedules.Source: D CASE STUDIES OF EXECUTED STRATEGIES · p. 35

D.1 INITIAL STRATEGY CONSTRUCTION GPT-6-Astra: a fixed linear predictor

  • A fixed linear-and-tree predictor produced identical forecasts under revision and the frozen control, with log gain 0.527 nats and top-1 29.6% for both.In the selected case, the same fitted program ran throughout days 1–10 while updating its race-specific feature inputs.Source: §D.1 · p. 35

D.2 CONFIDENCE ADJUSTMENT

  • For confidence adjustment, Log gain rose from 0.093 to 0.383 nats while Top-1 stayed at 25.3% (S →A).A changed 300 probability vectors but no complete ranking or top pick; mean top-pick probability fell from 53.9% to 29.3% on the same 324 races. Seven source snapshots contained three parsed code structures, with the final four changes adding comments only.Source: §D.2 · p. 36

D.3 FEATURE AND WEIGHT REVISION

  • The final DeepSeek-V4.1-Flash strategy used race-standardized weighted features, combining card, connection, workout, and recent-form signals.Its feature count varied by day, from 12 on day 1 to 25 on days 8–10, with recent form querying up to five previous starts.Source: §D.3 · p. 37
  • For DeepSeek-V4.1-Flash, Log gain fell from 0.288 to 0.230 nats and Top-1 from 22.5% to 19.8% (S →A).Seven source snapshots represented six parsed code structures, and the final two snapshots shared the same structure.Source: §D.3 · p. 37
  • Before each of days 2, 3, and 4, 27 of 28 weight entries changed relative to the preceding version; from day 5 onward, the weight vector stayed fixed.Both arms used the common 28-feature predictor for the first 24 races.Source: §D.3 · p. 37
  • Log gain rose from 0.471 to 0.486 nats and Top-1 from 27.8% to 30.2% for Claude Fable 5.1 (S →A).A changed 70 top picks and 298 complete rankings; probability vectors differed on 300 races, with predictor delivery on all 324 races despite ten provider failures.Source: §D.3 · p. 37

D.4 INFORMATION ACQUISITION

The acquisition workflow revises a shared initial evidence-gathering program, then uses acquired evidence for a fresh forecast on each race. Paired results show that lower fees can either raise or reduce net score, depending on the effect on forecast reward.

D.4 INFORMATION ACQUISITION

  • From Day 4, Adaptive R1 used a shorter card-only predictor, reduced data calls to one per prediction, and handled the remaining 252 races.The source shrank from 123 to 70 lines, and the revised version also changed prediction code.Source: §D.4 · p. 38
  • Net score fell from 30.51 to 16.07 score-yen per race as protocol fees fell by 3.53 but gross forecast reward fell by 17.97.Log gain declined from 0.333 to 0.187 nats, and Top-1 from 20.1% to 16.7%.Source: §D.4 · p. 38
  • Both arms began with the same acquisition program P0 and executed its initial query policy on Day 1.The first program read the race card and recent form for every prediction using two data calls.Source: §D.4 · p. 38
  • Adaptive R1 removed broad condition-statistics acquisition on Days 2–4, queried only the first listed runner on Day 5, then removed those queries again.One saved intermediate version was replaced before formal execution.Source: §D.4 · p. 38
  • Frozen-P0 made 3,500 native condition-statistics acquisitions and bought 1,136,800 cells, versus 287 acquisitions and 92,904 cells for Adaptive R1.Weighted condition-statistics charges were 6,559.00 and 536.27 score-yen, respectively; both arms also made 4,568 depth-one recent-history acquisitions and 324 race-level workout purchases.Source: §D.4 · p. 38
  • Across 324 races, Adaptive R1 raised net score from 1,892.94 to 7,266.70 score-yen versus Frozen-P0, while weighted fees fell from 12,609.19 to 6,586.46.Gross score moved from 14,502.13 to 13,853.16; the two arms generated forecasts separately.Source: §D.4 · p. 38

E Reproducibility and Released Materials

The release includes materials for inspecting the interface and selected scoring rules, alongside an aggregate audit of saved outputs. It does not provide all materials needed to reproduce the empirical results or rerun predictors.

E Reproducibility and Released Materials

  • The aggregate audit checks run IDs, pairing, completion, protocol and tariff identity, score arithmetic, delivery totals, replication counts, means, and day-level contrasts.Settlement is filtered by exact run ID, and source hashes guard against silent cohort refreshes; predictor re-execution and archive-level information-flow checks are separate tasks.Source: E Reproducibility and Released Materials · p. 39
  • Public release still requires a complete run-by-run audit-coverage manifest, immutable software/configuration metadata, releasable rendered prompts, and authorized archive-reconstruction instructions.The benchmark supports archive replay and sealed-program re-execution.Source: E Reproducibility and Released Materials · p. 39
  • The interface supplement includes the reference prompt, tool schemas, phase instructions, configuration, source-hash provenance, selected code excerpts, and tests using invented in-memory examples.Commercial archive records, private identifier mappings, and run conversations are excluded.Source: E Reproducibility and Released Materials · p. 39
  • Full numerical reproduction requires permitted analysis inputs and scripts, while predictor re-execution requires authorized archive access and run-specific artifacts.Aggregate arithmetic checks, predictor replay, and information-timing audits must be reported separately with their actual coverage.Source: E Reproducibility and Released Materials · p. 39

Related papers

Loading 2610.19589v1…