WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

• Experimental understanding is measured as the accuracy of quantitative predictions about interventions, represented by a complete response surface of conditional component effects.The agent inspects workflow code, selects experiments under a measurement budget, and submits predictions for unmeasured interventions.Source: §1 · pp. 2–3
• 35 of 36 task conditions contain interaction sign reversals, and 24 retain a reversal when both directions exceed 5% of the response range.A component change can improve one configuration and degrade another, so a preferred configuration or average effect does not capture all conditional behaviors.Source: §1 · p. 2
• The benchmark supports controlled comparisons of measurement selection, numerical inference, and program structure, including relations such as parameters becoming inactive when an operation is disabled.Shared-estimator comparisons hold observations fixed while changing inference; code-equivalence controls encode program rules as measurement and prediction constraints.Source: §1 · p. 2
• The benchmark spans 36 task conditions on 30 sources across eight workflow families, with 1,248 configuration records, 3,392 conditional effects, and 342 mean pair interactions.It provides standalone execution and grading.Source: §1 · p. 3
• Across 4,206 numerical-control records, 108 core episodes, eight structure-diagnostic episodes, and six added-family submissions, results separate optimization from intervention knowledge and show inference and executable equivalences improve reconstruction.Source: §1 · p. 3

2 RELATED WORK

• Scientific benchmarks test interaction (Jansen et al., 2024), experiment selection, prediction, or discovery (Koblischke et al., 2025; Gandhi et al., 2025; Duan et al., 2025), causal explanations (Chen et al., 2026), and scientific understanding (Rao et al., 2026); WhatWorkedBench measures complete conditional-effect predictions against exhaustive computational outcomes.Source: §2 · p. 3
• WhatWorkedBench delivers a complete numerical response surface, complementing benchmarks of experiment plans (Zhao et al., 2025; Abramovich and Chechik, 2025; Liu et al., 2026), research execution and outcomes (Huang et al., 2024; Chan et al., 2025; Gu et al., 2024; Chen et al., 2025; Garikaparthi et al., 2026), rediscovered findings (Wang et al., 2026), and reproduction traces (Hong et al., 2026).These benchmarks assess plans, execution, outcomes, findings, or traces rather than the complete numerical response surface identified as WhatWorkedBench’s knowledge artifact.Source: §2 · p. 3
• WhatWorkedBench’s native-verified code equivalences support error measurement for every conditional effect and comparisons of acquisition and inference using the same observations, linking agent evaluation to Bayesian experimental design (Rainforth et al., 2024) and hyperparameter importance.Conditional parameter spaces with inactive options are established in hyperparameter optimization (Lindauer et al., 2022).Source: §2 · p. 3

Task protocol

• Each task uses a deterministic utility table over four or six binary factors; agents receive two free anchor measurements and may buy at most B additional distinct measurements.Measurements can be queried adaptively or in batches, and all masks are legal.Source: §3 · p. 3

Submission protocol

• Agents submit finite predictions in [0, 1] for every legal mask, and grading replaces predictions at measured cells with authoritative observations.Submission closes the episode and fixes the delivered artifact used for final scoring.Source: §3 · p. 3

Evaluation target

• Complete conditional effects and one anchor determine the response surface, and errors bounded by ϵ per effect bound any k-component intervention error by kϵ.Effects are evaluated across every component and background configuration.Source: §3 · p. 4

Evaluation metrics

• Relative recovery is max(0, 1−Et/At), where Et is conditional-effect MAE and At is mean absolute true conditional effect; flat tables score one only when Et ≤ 10^-12.The benchmark also reports Et, At, and utility range to preserve reconstruction scale.Source: §3 · p. 4

Additional metrics

• Evaluation also reports pair-interaction MAE, grid MAE, and selection regret; strict reconstruction requires every effect error to stay within max(10^-12, τ times utility range).The development tolerance is τ = 0.1, and evaluation includes a five-tolerance profile alongside continuous recovery.Source: §3 · p. 4

Catalog

• The catalog covers 36 task conditions on 30 sources, with 352 four-factor and 896 six-factor configuration records; 96 reproduce original subcube outcomes.A paired track adds six six-factor variants on existing sources; the six-factor track also includes beat-detection and graph instances.Source: §4 · p. 4, Table 1 · p. 4

Conditional structure across the catalog

• At a 10^-12 floor, 35 of 36 conditions show sign reversals; requiring both signs above 1%, 5%, or 10% of response range leaves 33, 24, and 16 conditions.Flat exports provide all 3,392 conditional effects and 342 mean pair interactions for reuse.Source: Figure 3 · p. 5, §4 · p. 5

Native evaluation

• Outcomes use documented native-library workflows with recorded sources, implementations, seeds, and cohorts, while evaluation preserves task-specific data-access and scoring boundaries.Protocols include training-only fitting, grouped feature-identical rows across splits, forecasts from preceding observations, clean restoration references, original retrieval annotations, and disjoint test pairs for link prediction.Source: §4 · p. 5

Reference validation

• Cold execution checks configuration predictions against recorded hashes, and the standalone episode engine agrees with 4,206 numerical-control records.The prepared-input pack replays all 1,248 records outside the project in 127.87 seconds using one computation thread; independent native metrics and target-perturbation checks verify scoring boundaries.Source: §4 · p. 5

Measurement credits

• Measurement credits impose an information budget: agents receive cached native outcomes for queried configurations, while exhaustive outcomes remain with the evaluator.Repeated queries to the same configuration return identical feedback, enabling comparisons against a complete numerical target.Source: §4 · pp. 5–6

5 EXPERIMENTAL SETUP

• Numerical controls compare additive and pairwise ridge models, sequential and random designs, and Gaussian-process baselines and measurement-selection strategies.The Gaussian process uses option differences to model similarity; Grid GP targets score uncertainty, while Effect GP targets conditional-effect uncertainty. All predictors retain measured values.Source: §5 · p. 6
• Public-code equivalence controls measure one representative per class of configurations with identical native behavior and assign its prediction to every class member.Equivalences are checked against native prediction hashes and scores; trajectories distinguish purchased measurements from inferred aliases.Source: §5 · p. 6
• The 108-episode core agent study covers the first source in each of six original workflow families; four-factor Flash and Pro evaluation spans 72 episodes across original and additional cohorts at B = 4 and 8.Agents received workflow code and input summaries, high-effort reasoning, a 32,768-token output cap, a 1,800-second deadline, and tools for evidence, purchases, Python calculations, and prediction submission.Source: §5 · p. 6
• The information study compares full workflow information with an opaque masks-and-utilities view in 36 Flash episodes across six paired six-factor variants at B = 20.Six complete Flash submissions on added beat-detection and graph tasks at B = 32 support a separate completed-artifact analysis; dated cohorts are reported separately.Source: §5 · p. 6
• Family-macro averaging weights workflow families equally, and missing delivery remains in the denominator with zero recovery.Four-factor evaluation delivered 63 of 72 artifacts; valid delivery is tracked independently of HTTP status.Source: §5 · p. 6

6.1 OPTIMIZATION SUCCESS AND INTERVENTION KNOWLEDGE ARE DISTINCT TARGETS

• At B = 8 on 22 four-factor sources, pair-effect design/ridge achieves 0.612 recovery and 15 exact choices, while effect-variance GP achieves 0.701 and 16.Pair ridge has three strict reconstructions versus one for GP, despite 13 pair-ridge instances combining an exact choice with effect error above the strict tolerance; pair ridge wins the maximum-error criterion while GP ranks higher on mean recovery and configuration choice.Source: Figure 4 · p. 7, §6.1 · p. 7
• Pair ridge passes maximum-edge tolerances of 2.5%, 5%, 10%, 20%, and 40% on 1, 1, 3, 8, and 20 of 22 instances, respectively.Among its 15 exact-choice cases, mean continuous recovery is 0.634 (minimum 0.361); at four factors and B = 8, native-scale recovery is 0.633 for pair ridge and 0.712 for effect-variance GP.Source: §6.1 · p. 7

6.2 SHARED INFERENCE EXTRACTS ADDITIONAL VALUE FROM ACQUIRED EVIDENCE

• Original Flash recovery rises from 0.178 at B = 4 to 0.632 at B = 8, while additional-cohort recovery rises from 0.375 to 0.621.Pro recovery rises from 0.043 to 0.366 in the original cohort and from 0.308 to 0.503 in the additional cohort; across 18 original Flash B = 8 episodes, seven select an exact optimum and one passes strict reconstruction.Source: §6.2 · p. 7, Table 2 · p. 8
• Shared GP fitting raises Flash recovery from 0.632 to 0.698 in the original cohort and from 0.621 to 0.720 in the additional cohort.The shared-estimator protocol holds the acquired observations fixed while changing the inference calculation; the GP gains remain positive when each original-cohort source is omitted in turn.Source: Table 3 · p. 8, §6.2 · p. 8
• For original Flash B = 8 trials, mean-completed unmeasured utilities yield 0.287 recovery versus 0.632 delivered, while 67.4% of conditional effects involve an unmeasured configuration.Mean effect error on remaining contrasts, normalized by At, is 1.054 for mean completion and 0.542 for agent artifacts.Source: §6.2 · p. 8

6.3 PROGRAM STRUCTURE INCREASES THE VALUE OF MEASUREMENTS

• At B = 20 across six paired workflows, code-equivalence classes raise pair-ridge recovery from 0.157 to 0.389 and effect-GP recovery from 0.248 to 0.462.At B = 32, pair-ridge rises from 0.445 to 0.650 and effect-GP from 0.537 to 0.697; all four sources with nontrivial classes improve, while two identity controls retain their predictions.Source: §6.3 · pp. 8–9
• Supplied rules change mean recovery from 0.247 to 0.429, while post-hoc projection raises recovery from 0.338 to 0.507 across eight B = 20 episodes.All eight final tables violate the equivalences despite code-only agents reporting all six registered rules before measurement and rules-supplied agents reporting them as well; full-code versus opaque recovery is 0.268 versus 0.257 in the original cohort and 0.185 versus 0.234 in the additional cohort.Source: §6.3 · p. 9

6.4 SHARED INFERENCE ON ADDED WORKFLOW FAMILIES

• On six complete B = 32 Flash submissions, shared GP raises family-macro effect recovery from 0.303 to 0.455, with gains on all six sources.The submissions cover beat detection and graph link prediction, three sources per family; same-source numerical controls reach 0.230 for D2, 0.414 for grid GP, and 0.404 for effect GP.Source: §6.4 · p. 9

7 A PLATFORM FOR RESEARCH ON EXPERIMENTAL UNDERSTANDING

• WhatWorkedBench supports experimental agents, adaptive design, response-surface inference, and program-based inference using public tasks, native references, observation traces, and exact effect tables.These resources support matched comparisons of measurement policies, estimators, and code-based constraints through the companion data guide.Source: §7 · p. 9

8 CONCLUSION

• The catalog covers 36 task conditions on 30 sources across eight workflow families, with 1,248 configuration records and exhaustive CPU references for conditional effects and interactions.Agents inspect workflow code, choose measurements, and submit a complete predicted response surface; shared-estimator comparisons separate measurement selection from numerical inference.Source: §8 · p. 9
• With fixed observations, shared GP raised added-family recovery from 0.303 to 0.455; at B = 20, code equivalences raised six-factor GP recovery from 0.248 to 0.462.The shared-estimator result covers all six completed added-family submissions.Source: §8 · p. 9
• In eight diagnostic episodes, post-hoc class projection raised recovery from 0.338 to 0.507 after agents reported program rules but submitted tables that violated them.The conclusion also reports that shared GP improves recovery on the same Flash observations in both dated API cohorts.Source: §8 · p. 9
• Across 4,206 numerical controls and 108 core agent episodes, optimal configuration choices coexisted with substantial conditional-effect error.The benchmark evaluates effect reconstruction, configuration choice, and artifact delivery.Source: §8 · p. 9

REPRODUCIBILITY STATEMENT

• The task pack’s standard-library engine reproduces 4,206 stored score records in a standalone process.The artifacts also include evaluator-only references, versioned episode semantics, exact native operations, numerical controls, and machine-readable audits.Source: REPRODUCIBILITY STATEMENT · p. 9
• The prepared-input pack reproduces all 1,248 native records outside the repository.Prepared inputs are included among the accompanying artifacts.Source: REPRODUCIBILITY STATEMENT · p. 9
• Source acquisition and preparation retain separate manifests and code identities.Source: REPRODUCIBILITY STATEMENT · p. 9

AI USE STATEMENT

• An AI coding assistant supported topic exploration, workflow design, protocols, data preparation, implementation, analysis, figures, and drafting; DeepSeek models served as evaluated agents.The study preserved generated implementations, analysis scripts, experiment plans, and numerical audit results; verification used cold native replay, separately implemented numerical metrics, and recorded source and code identities.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The work uses public research archives under their source terms, with ECG tasks evaluating event detection on archived waveforms and scoring references held by a trusted host.Agents receive task descriptions and budgeted aggregate observations.Source: ETHICS STATEMENT · p. 10

A BENCHMARK DESIGN AND RELATED EVALUATIONS

• WhatWorkedBench grades the full conditional-effect field implied by a response surface and compares measurement policies under shared estimators and estimators on the same observations.Its reference uses native metric outcomes for documented component interventions.Source: A BENCHMARK DESIGN AND RELATED… · p. 13

B SOURCE INVENTORY

• The catalog covers 30 sources and 36 task conditions; paired four- and six-factor variants reproduce the original four-factor outcomes in 96 repeated configuration records.Source-level reporting merges paired variants when counting distinct sources.Source: B SOURCE INVENTORY · pp. 14–15

C.1 TABULAR CLASSIFICATION, REGRESSION, AND CLUSTERING

• Tabular evaluation fixes grouped train/evaluation splits across masks and fits transforms on training data, while forecasting uses a fixed training fit with realized prior observations for each one-step prediction.The tabular evaluation fraction is 0.3; forecasting training starts at index 12 for every mask.Source: §C.1 · p. 16, §C.2 · p. 16

D WORKFLOW OPTIONS AND FIXED NUMERICAL SETTINGS

• The workflow tracks define binary masks over four or six options, with every mask legal and option settings, operation order, and numerical details specified in the tables.Public task files provide the corresponding executable operation definitions.Source: D WORKFLOW OPTIONS AND FIXED NUMERICAL… · p. 17

E RESPONSE SCALES AND CONDITIONAL STRUCTURE

• The response-scale summary reports score ranges and conditional-effect magnitudes by workflow family; At is the mean absolute conditional effect used to normalize recovery.Each interval spans task conditions within its family.Source: E RESPONSE SCALES AND CONDITIONAL… · p. 19

F.1 COVERAGE OF JOINT INTERVENTIONS

• Exact conditional effects determine every joint intervention and, with one anchor, every utility; a k-switch intervention’s error is bounded by kϵ when maximum edge error is at most ϵ.Mean edge MAE measures average reconstruction quality, and higher-order interactions remain represented in the background dependence of conditional effects.Source: §F.1 · pp. 19–20

F.2 CONFIGURATION CHOICE AND EFFECT RECOVERY

• For 0 < δ < 1, the two-bit examples select 10 as the unique optimum while effect recovery is R = δ, so exact choice can coexist with arbitrarily weak recovery.The construction extends to four- and six-factor spaces by adding inactive bits.Source: §F.2 · p. 20

F.3 NATIVE METRICS AND UTILITY TRANSFORMATIONS; F.4 EFFECT, INTERACTION, AND GRID ERRORS; F.5 ARTIFACT VALIDATION AND THE STRICT DIAGNOSTIC

• Balanced accuracy, NMI, NDCG, event F1, and ROC AUC directly supply utility; a complete artifact must provide exactly one finite numerical utility in [0, 1] for every legal mask.Selection breaks utility ties by fewer enabled bits, then the lexically smaller mask; exact choice allows regret at most 10^-12. Missing delivery receives zero recovery and zero exact-choice credit.Source: §F.3 · p. 20, §F.5 · p. 21, §F.4 · p. 21

F.6 MEASURED EFFECTS AND REMAINING RECONSTRUCTION

• At B = 8, shared GP recovery is 0.698 versus 0.632 for delivered tables and 0.287 for mean completion in the original Flash cohort, with 67.4% of effects requiring reconstruction.The comparison uses the same observations and restores measured cells before scoring.Source: §F.6 · p. 21

G.1 MAIN-EFFECT AND PAIR-EFFECT RIDGE; G.2 GAUSSIAN-PROCESS CONTROLS; G.3 PROGRAM-EQUIVALENCE CONTROLS; G.4 SHARED ESTIMATORS AND AGGREGATION

• Degree-one and degree-two ridge use centered-bit features, while GP controls use an all-order Hamming kernel; code-equivalence controls fit representatives, and shared estimators evaluate on the episode’s exact observed set.The ridge regularizer is chosen by leave-one-out error; GP acquisition targets grid or effect variance, and aliases are validated against matching native prediction hashes and utilities.Source: §G.1 · p. 22, §G.2 · p. 22, §G.3 · p. 22

H AGENT INTERFACE AND EXECUTION ENVIRONMENT; I RECORDED PROMPTS AND TASK PAYLOADS; J STUDY DESIGN AND EXECUTION ACCOUNTING; K EPISODE RESULTS AND PAIRED COMPARISONS

• The study provides tools for batch observations, numerical calculation, and complete-table or estimator submission, and records prompts, budgets, cohort execution, and episode accounting.The 4,206 records include ridge/design, code-equivalence ridge, GP, and added-source runs; four-factor budgets span 0–14 measurements and paired six-factor budgets span 0–62.Source: H AGENT INTERFACE AND EXECUTION… · p. 23, I RECORDED PROMPTS AND TASK PAYLOADS · p. 24, J STUDY DESIGN AND EXECUTION ACCOUNTING · p. 24

K.1 INFORMATION SUPPLIED TO THE AGENT

• Full-minus-opaque recovery differences are 0.0103 in the original cohort and −0.0491 in the additional cohort; among eight additional delivered pairs, source-macro differences are −0.1198.The corresponding shared-GP differences are −0.0242 and −0.0638 across all attempts, and −0.0914 for the eight-pair source-macro summary.Source: §K.1 · p. 25

K.2 REPEATED HIGH-BUDGET AGENT RUNS

• Original Flash recovery is 0.6321 (shared GP 0.6976), while additional-cohort recovery is 0.6211 (shared GP 0.7205); five original episodes select an exact optimum and one passes strict reconstruction.In the six additional episodes, two select an exact optimum and none pass strict reconstruction.Source: §K.2 · p. 26

K.3 STRUCTURE RECOGNITION AND APPLICATION

• At B = 20, supplied rules raise arm-mean recovery from 0.247 to 0.429, while post-hoc equivalence projection raises mean recovery from 0.338 to 0.507 with fixed observations.Both arms correctly report all six registered rules, but all eight submissions violate equivalences; projection improves recovery in seven and leaves strict score at zero.Source: §K.3 · p. 26

L.1 SENSITIVITY TO INDIVIDUAL SOURCES

• The original shared-GP mean advantage remains positive under every source omission, ranging from 0.021 to 0.096; the additional cohort’s gain is 0.099, with an omission range of 0.069 to 0.120.Four sources improve and two decline in each cohort; program-equivalence controls improve on all four sources with nontrivial classes.Source: §L.1 · p. 27

L.2 STRICT-ERROR TOLERANCES

• At B = 8, D2 passes the 10% strict-error tolerance on 3/22 sources and the 20% tolerance on 8/22, while its 15 exact choices remain unchanged.Strict counts use maximum conditional-effect error relative to response range; exact counts are optimal configuration choices.Source: §L.2 · p. 27

L.3 NATIVE-UNIT SENSITIVITY

• At four factors and B = 8, D2 recovery is 0.612 in utility units and 0.633 in native units; GP-E recovery is 0.701 and 0.712, respectively.The native-unit analysis applies the inverse utility map while keeping fitted models and observed sets fixed.Source: §L.3 · p. 28

L.4 MAGNITUDE OF CONDITIONAL REVERSALS

• A conditional reversal requires positive and negative background effects whose magnitudes both exceed the specified floor.Source counts merge paired dimensional variants, and the two largest floors identify substantial conditional changes within the fixed response range.Source: §L.4 · p. 28

L.5 SIGNAL AND GRAPH WORKFLOWS

• The eight added six-factor cases cover beat-detection and link-prediction workflows, with order ≥3 variance measuring the share of factorial variance carried by interactions of order three and above.Curves average random seeds within each source and then four sources within each family.Source: §L.5 · p. 28
• At B = 20, two-family recovery is 0.0439 for pair ridge and 0.0435 for GP-E.The larger-budget trajectories and exhaustive endpoints expose the remaining reconstruction problem under sparse measurements. Figure 5 shows recovery across budgets for beat detection and link prediction.Source: Figure 5 · p. 29, §L.5 · p. 29

L.6 COMPLETED AGENT PREDICTIONS ON THE ADDED FAMILIES

• Six Flash episodes at B = 32 supplied complete tables for three beat-detection and three graph-link prediction sources, compared with a GP fit to the same purchased observations and numerical controls.The interface used two free anchors and session limits matching the six-factor agent interface; each family row averages three submitted sources.Source: §L.6 · p. 29
• Family-macro recovery rises from 0.303 to 0.455 with a shared GP on the six completed beat-detection and graph submissions.The corresponding beat and graph gains are 0.160 and 0.145; scores condition on complete final artifacts.Source: §L.6 · p. 30