PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li

arXiv:2610.00314v1cs.AI

TL;DR

The paper asks what scientific explanations add to forecasts of planned experiments. It measures that contribution with paired forecasts across description, matched, and donor contexts. Its frozen v5 decision was inconclusive, so predictive credit remained unconfirmed in the tested studies.

  • Problem

    Research-agent systems produce both experimental results and explanations, but evaluating an explanation requires measuring what it adds to predictions of those results.

    Source: §1 · p. 2

  • Method

    Paired forecasts of fixed interventions compare description, matched-explanation, and donor contexts while tracking commitment, delivery, predictive value, alignment, and known-signal sensitivity.

    Source: §3.3 · p. 4, §9 · p. 9

  • Results

    The frozen v5 decision was inconclusive; Tox21’s primary ROC AUC interval-score harm rule was unmet, and OpenML returned confirmation no go.

    Source: §9 · p. 9

  • Takeaways & Limitations

    Forecast-based ranking requires demonstrated predictive gain and a task-specific cost-utility rule.

    Source: §9 · p. 9

    Natural-explanation predictive credit remained unconfirmed in the tested frozen studies.

    Source: §1 · p. 2

Abstract

from arXiv · show

Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake. Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, v5's frozen credit decision was inconclusive. Tox21's preregistered ROC AUC interval-score harm test was unmet ($D-M=-.0026$, 95 percent interval [$-.0174$, .0104]); OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero. Under requested DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5 and 59.1 percent. A DeepSeek V4 Flash replay raised matched point MAE from .01823 to .02020 and missed matched-donor interval-score equivalence. OpenML full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage versus 51.4 percent for description and content in 66/144 cards. Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain. A researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description. The protocol measures predictive credit for research-agent benchmarks and scientific forecasting; natural-explanation credit remained unconfirmed at the tested donor resolutions.

1 INTRODUCTION

The paper asks how much an agent’s scientific explanation adds to forecasts of an experiment’s outcome. It measures this with paired forecasts and reports that predictive credit remained unconfirmed in its frozen studies.

1 INTRODUCTION

  • An explanation’s practical value lies in helping anticipate how an intervention will affect an outcome, informing experimental resource allocation.For example, a rationale for stronger regularization can predict effects on generalization or the training gap.Source: §1 · p. 2
  • Scoring explanations alongside achieved results gives research-agent benchmarks a way to evaluate the rationale as well as the search outcome.The paper’s contributions include paired evaluation, prospective and component controls, direct-text forecasts, and known-signal sensitivity tests.Source: §1 · p. 2
  • Paired forecasts compare description, matched explanation, and donor contexts for the same outcome, forecaster, and intervention.The donor context tests whether gains depend on matching the explanation to the current case.Source: §1 · p. 2
  • The frozen studies left predictive credit unconfirmed; Tox21’s primary ROC AUC interval-score harm rule was unmet.The study also examines repeat drift, component controls, and direct-text forecasts.Source: §1 · p. 2

2 RELATED WORK

Prior work forecasts scientific outcomes, tests whether explanations support prediction, and studies research-agent performance and uncertainty. This paper connects those lines by measuring how an explanation changes forecasts of executed experiments.

2 RELATED WORK

  • Scientific forecasting includes pairwise predictions of empirical AI outcomes (Wen et al., 2025), neuroscience findings from study descriptions (Luo et al., 2025), and outcome-based research-idea comparisons (Mule et al., 2026).Research preference models select experiments using plans, code, and optional pilot runs (Foster et al., 2026); CUSP evaluates scientific advances under temporal knowledge constraints (Wu et al., 2026).Source: §2 · p. 2
  • Simulatability work tests whether explanations help predict model outputs (Chen et al., 2024), while this paper forecasts numerical changes in external experiments.Other work examines information from activations under counterfactual prompt edits (Karvonen et al., 2026).Source: §2 · p. 2
  • Reasoning-trace interventions and revision studies separate text’s influence from recomputation and repeat variability (Lanham et al., 2023; Ning & Li, 2026).This paper repeats forecasts for one fixed experiment and reports both drift and predictive error.Source: §2 · p. 3
  • Research-agent evaluation spans task performance, evidence uptake, and falsifiable causal predictions (Bragg et al., 2026; Ríos-García et al., 2026; Zhao et al., 2026).The paired design measures the gain from a supplied explanation for fixed executed interventions.Source: §2 · p. 3
  • Measurement work distinguishes observed measures from intended constructs and studies calibration, self-consistency, and semantic uncertainty; this paper measures drift, point error, and interval score.The proper interval score is attributed to Gneiting & Raftery (2007).Source: §2 · p. 3

3 EVALUATING SCIENTIFIC EXPLANATIONS

The evaluation compares forecasts for fixed interventions across description, matched-explanation, and donor contexts. It tracks what explanations commit to, whether their content is delivered, and whether they improve forecasts or respond to known signals.

3.1 PREDICTIVE VALUE AND LOCAL ALIGNMENT

  • Description, matched explanation, and frozen donor contexts forecast the same held-out child-minus-parent effect for a fixed state and intervention.Public state and planned intervention define the description; the donor explanation is assigned from another state.Source: §3.1 · p. 3
  • Across 72 states, matched/donor point MAE was .02041/.02023 in Figure 2’s Tox21 comparison.The figure links one Tox21 pair’s claims and executed outcomes; its example is NR-AhR at 40% labels, with fingerprint radius increased from 2 to 3.Source: Figure 2 · p. 4

3.2 COMMITMENT, AGREEMENT, AND PREDICTION

  • Commitment records six testable prediction-card slots; agreement measures variation across two calls, while prediction reports point error, direction, interval coverage, width, and interval score.The six slots are target direction, target point or interval, intermediate observable, benefit regime, falsification criterion, and counterfactual.Source: §3.2 · p. 4

3.3 FIVE CHECKS FOR PREDICTIVE CREDIT

  • Five checks cover commitment, evidence-backed delivery, paired predictive value, donor alignment, and known-signal sensitivity; task-aware aggregation treats task as the top inference unit in external studies.Intention-to-treat analysis retains all assigned calls, including invalid responses handled by deterministic fallback; delivered-card comparisons describe selected cases where content arrived.Source: §3.3 · p. 4

4 EXPERIMENTAL DESIGN

The prospective program tested three settings and targeted follow-ups, with frozen source-study rules for predictive value, interval-score harm, and a joint OpenML criterion. V5 was inconclusive, while Tox21 and OpenML returned confirmation no-go.

4.1 THREE EXPERIMENTAL SETTINGS

  • The prospective studies covered 120 controlled-learning states, 72 Tox21 states, and 144 OpenML states; the follow-up mechanism positive control added 40 derived states.Tox21 used 12 endpoints, three label fractions, and two seeds; OpenML sampled 12 classification and 12 regression tasks, each with three budgets and two seeds.Source: §4.1 · p. 5
  • The Tox21 frozen harm gate required M−D and M−S to reach .005 with positive one-sided lower bounds, but returned confirmation no-go.Table 1 presents selected frozen source-study contrasts; its intervals are 95% for v5 and Tox21 and 90% for OpenML.Source: §4.1 · p. 5, Table 1 · p. 5

4.2 TARGETED MODEL REPLAY AND CHANNEL CALIBRATION

  • Targeted follow-ups reused source states for the 432-call Tox21 replay and 1,440-call calibration; the mechanism control added 40 paired states.Positive-control outcomes were computed before prompt freeze and withheld from the forecaster.Source: §4.2 · p. 6

4.3 EVIDENCE STATUS AND STATISTICAL ANALYSIS

  • The frozen v5 rule was inconclusive, and the Tox21 harm and OpenML joint criteria both returned confirmation no-go.Cross-study analyses and controls used existing outcomes under separately frozen call plans.Source: §4.3 · p. 6
  • Inference used endpoints and tasks as external units, with endpoint, budget-cell, and seed bootstraps for Tox21 and task-stratified resampling for OpenML.The post-outcome crossover froze its fixed-number mechanism contrast before Flash calls; other arm contrasts were exploratory and unadjusted for multiplicity.Source: §4.3 · p. 6

5 SEPARATING REPEATABILITY FROM PREDICTIVE GAIN

Cards reduced repeated-forecast drift in Tox21, but the frozen harm test and Flash replay did not confirm predictive credit. Other follow-ups likewise showed no confirmed matched point-accuracy advantage.

5 SEPARATING REPEATABILITY FROM PREDICTIVE GAIN

  • The frozen Tox21 ROC AUC interval-score harm rule returned confirmation no-go; Pro repeat drift fell 64.5% with matched cards and 59.1% with seed-level donors.Point MAE was .02056 under description, .02041 with matched cards, and .02023 with donors.Source: §5 · p. 6
  • In the 432-call Flash replay, matched and donor cards reduced drift by 60.6% and 59.2%, but matched-card point MAE rose from .01823 under description to .02020.The frozen joint status was replication not supported because interval-score equivalence exceeded its ±.010 margin.Source: §5 · p. 6
  • Across the 1,152-call crossover, all seven card contexts had point MAE above description’s .01738, ranging from .01797 to .01873.At fixed numbers, donor-minus-matched screened-mechanism point MAE was −.00047 with interval [−.00228, .00104].Source: §5 · p. 6
  • OpenML Pro mean drift was .01004 under description and .01300 with matched cards; the description-minus-matched difference was −.00296 with 90% interval [−.01212, .00254].The 10% trimmed drift values were .00673 and .00610, respectively.Source: §5 · p. 6
  • Direct-text Flash mean drift was .00488 under description and .00962 under matched notes.OpenML reports ROC AUC-change and skill-change units, respectively.Source: §5 · p. 7

6 COMMITMENT AND UNCERTAINTY

The section separates elicited commitments and directional prediction from uncertainty calibration. Cards increased completion in controlled v5, while OpenML intervals widened despite coverage below the nominal target.

6.1 ELICITATION CHANGES COMMITMENT RATES

  • Controlled v5 produced complete cards in 0 of 60 ordinary proposals and 59 of 60 elicited proposals; OpenML completeness was 1 of 144 versus 20 of 144.The OpenML difference was 13.2 percentage points with a one-sided 95% lower bound of 6.9 points; the comparison combines interface, effort, schema, and extraction differences.Source: §6.1 · p. 7

6.2 A SIGN PRIOR EXPLAINS MUCH OF DIRECTION ACCURACY

  • In v5, direction accuracy conditional on a claim was 59.6% under ordinary prompting and 57.6% under elicitation; elicited central-80-percent card intervals covered 33 of 59 outcomes (55.9%).The 68.6% figure is described as a post hoc movement-sensitivity comparison, with the full threshold sweep in the appendix.Source: §6.1 · p. 7, §6.2 · p. 8

6.3 WIDER INTERVALS COEXIST WITH UNDERCOVERAGE

  • OpenML full-card assignment increased mean interval width from .06179 to .07474, about 21%, while coverage was 49.3% with full cards and 51.4% with description.The paired width change was .01294 with task-stratified 95% interval [.00141, .03306]; coverage difference was −2.08 percentage points with interval [−6.25, +2.08].Source: §6.3 · p. 8
  • Across OpenML contexts, forecast intervals covered 45.8% to 53.5% of outcomes against a nominal 80% target; v5 also showed width growth with only small coverage changes.For v5, accuracy interval width grew from 4.63 to 9.37 percentage points as coverage moved from 59.2% to 61.3%; log-loss width grew from .295 to 2.796 as coverage moved from 48.8% to 51.3%.Source: §6.3 · p. 8

7 DELIVERY, SCALE, AND DETECTABLE INFORMATION

The delivery, scale, and known-information checks show that measured explanation value depends on the forecast pipeline and task population. Direct-text Flash showed no detectable matched point-accuracy advantage, while the mechanism positive control improved point error.

7.1 DELIVERY DEFINES THE CONTENT COMPARISON

  • Direct-text Flash delivered all 144 notes and showed no detectable matched point-accuracy advantage; point MAE was .08257 for description, .08311 for matched, and .08197 for donor.Task-balanced scaled matched gains against description and donor were −.0906 and .0064, with 95% intervals [−.2687, .0403] and [−.0964, .0898].Source: §7.1 · p. 8

7.2 TASK SCALE DETERMINES INFLUENCE ON THE AGGREGATE

  • The six brazilian houses states accounted for 37.0% of description error and 41.1% of matched-full error; excluding them, description MAE was .05498 and matched-full MAE was .05614.Overall description ensemble MAE was .08368 against a .08341 zero forecast; without brazilian houses, the zero baseline was .05910.Source: §7.2 · p. 9
  • On the original scale, description-minus-full MAE was −.00741 with 90% interval [−.02804, .00251], and within-shuffle-minus-full was −.00731 with interval [−.03749, .00654].Both intervals extend below −.01.Source: §7.2 · p. 9

7.3 POSITIVE CONTROLS FOR KNOWN INFORMATION

  • The researcher-authored true mechanism note lowered point MAE to 3.795 from 6.391 under description and 8.151 under a false note; its gain over description was 2.596 percentage points.The gain’s 95% pair-bootstrap interval was [2.314, 2.880]; the exact-signal calibration separately had 144/144 wins and Spearman correlation .99996.Source: §7.3 · p. 9

8 APPLICATIONS IN RESEARCH EVALUATION

The protocol evaluates explanations through five checks, including predictive credit and information uptake, and proposes forecast-based ranking only when paired predictive checks and cost-utility criteria are met.

8 APPLICATIONS IN RESEARCH EVALUATION

  • Forecast-based proposal ranking requires demonstrated predictive gain and a task-specific cost-utility rule.The protocol measures explanations on executed experiments.Source: §8 · p. 9
  • The five checks cover elicited-card completeness, content delivery, predictive credit, seed-donor alignment, and exact-signal uptake.OpenML’s completeness gain was .132, below its frozen .50 formation threshold; content appeared in 66/144 cards.Source: §8 · p. 9
  • The joint predictive-credit result remained unconfirmed, and seed-donor intervals spanned zero in Tox21 and OpenML.The assay-sensitive exact-signal control was positive.Source: §8 · p. 9

9 CONCLUSION

The conclusion reports unconfirmed predictive credit across the prospective studies, while the appendix documents study designs, controls, data handling, and analysis records. The work frames its protocol as a way to evaluate research-agent forecasts and supplied information.

9 CONCLUSION

  • The v5 report found no confirmed credit, and Tox21 and OpenML returned confirmation no go; the conclusion also calls out researcher-authored mechanism and exact-signal controls.The source studies froze records before outcomes; cross-study analyses and controls used known outcomes under separately frozen call plans.Source: §9 · p. 9, §A.1 · p. 13

ETHICS STATEMENT

A.2 CONTROLLED V5 DESIGN

  • The 120-state controlled v5 design crossed six learning regimes and frozen seeds, with spontaneous or elicited notes and two forecasts in each of three contexts.The study also executed a prespecified follow-up counterfactual in 30 cases.Source: §A.2 · p. 13

A.3 TOX21 ANCHOR DESIGN

  • Tox21 crossed 12 endpoints, three label budgets, and two seeds, with an action assigned to each endpoint-budget cell.Both seeds in a cell received the same action; the frozen harm decision used endpoint-, budget-cell-, and seed-level resampling.Source: §A.3 · pp. 13–14

A.4 OPENML V2.1 DESIGN

  • OpenML confirmation tasks came from distinct source families, and the study used grouped splits, three training-label budgets, two seeds, and six assigned interventions.Forecasts targeted private held-out skill change before measured child validation results were supplied.Source: §A.4 · p. 14

A.5 ASSAY CALIBRATION DESIGN

  • The assay calibration compared exact, noisy, and shuffled outcome signals against prespecified accuracy, correlation, and integrity gates.The exact signal had to beat both description and wrong-seed signal by the frozen thresholds.Source: §A.5 · p. 15

B.1 FORMATION AND DIRECTION

  • Direction forecasts were near an always-improvement baseline, while the binary search-prediction correlation did not identify the intended separation.Direction accuracy was 58.6 percent versus 57.7 percent for always predicting improvement; point forecasts correlated .4964 with outcomes.Source: §B.1 · p. 15

C.1 FROZEN DECISION

  • The Tox21 frozen harm criteria were not met: both estimates fell below .005, endpoint and action counts missed thresholds, and contrasts changed sign across budgets.The required positive contrasts covered at least eight endpoints and four actions.Source: §C.1 · p. 16, Table 10 · p. 17

C.2 ALL CONDITION METRICS

The supplied passages identify Tox21 condition metrics, calibration, secondary contrasts, and endpoint-, action-, and budget-specific diagnostics, but provide only limited numerical results for these subsections.

C.2 ALL CONDITION METRICS

  • Tables 11–13 cover Tox21 condition metrics, prospective-card calibration, and secondary matched-card contrasts.The supplied table captions identify scope but do not provide their cell values.Source: Table 11 · p. 17, Table 12 · p. 17, Table 13 · p. 17

C.3 ENDPOINT, ACTION, AND BUDGET DIAGNOSTICS

  • All 72 Tox21 cards, 432 forecasts, and 72 outcomes were present, with all eight required integrity strata at 1.0.Four first attempts failed in transport and succeeded on their single frozen retry.Source: §C.3 · p. 17
  • The endpoint- and action-specific tables report Tox21 interval-score harm across endpoints, actions, and budgets.Their captions establish the diagnostic scope, not the table results.Source: Table 14 · p. 18, Table 15 · p. 18

D FLASH REPLAY AND EQUIVALENCE CHECKS

The Flash replay reproduced lower repeat drift but did not establish interval-score equivalence. OpenML results likewise show widened intervals and no confirmed point-accuracy or repeatability gains, with task scale and delivery affecting interpretation.

D FLASH REPLAY AND EQUIVALENCE CHECKS

  • Matched and shuffled Flash cards reduced Tox21 repeat drift by 60.55 and 59.17 percent, but the frozen joint replay status was replication not supported.The sole failed gate was interval-score equivalence: matched minus shuffled was .003268 with 95 percent interval [−.008096, .014485], extending beyond the ±.010 margin.Source: D FLASH REPLAY AND EQUIVALENCE CHECKS · p. 18, Table 16 · p. 18

E.1 POPULATION AND INTERVENTION OUTCOMES

  • The 144 OpenML states had 90 positive and 54 negative intervention effects, with median absolute skill change .0236264.The final population included 144 route states, notes, extractions, forecasts, and outcomes.Source: §E.1 · p. 19

E.2 FORMATION AND DELIVERY

  • OpenML’s equal-task complete-card difference was .131944, below the frozen .50 formation requirement.The one-sided 95 percent lower bound was .069444, and 15 of 24 tasks were positive.Source: §E.2 · p. 19, Table 17 · p. 19

E.3 CONDITION-LEVEL FORECASTS

  • Under the frozen OpenML call-level convention, full-card minus description width was +0.0129432 (task-stratified 95 percent interval [+0.0014148, +0.0330630]); coverage difference was −0.0208333 (interval [−0.0625, +0.0208333]).Source: §E.3 · p. 19

E.4 FROZEN POINT AND REPEATABILITY CONTRASTS

  • OpenML’s frozen point-equivalence and repeatability criteria were unmet; its mean full-card drift was .01300 versus .01004 for description.The hierarchical intervals extended beyond the ±.01 point-equivalence margin, and the prespecified repeatability conditions failed.Source: §E.4 · p. 20, Table 21 · p. 20

E.5 SCALE AND NULL-BASELINE AUDIT

  • The same OpenML regression task was worst in every condition, and taskwise mean MAE ratios exceeded 1 for most contexts.The means exceeded medians, indicating high-error task influence and motivating taskwise reporting alongside pooled error.Source: §E.5 · p. 20, Table 22 · p. 20

E.6 POST HOC MOVEMENT SWEEP

  • In the post hoc movement sweep, interval coverage decreased as realized outcome movement grew.Figure 5 compares call-level coverage across outcome-size subsets and task-level changes in pooled ensemble MAE after omitting each task; these are descriptive comparisons.Source: §E.6 · p. 21, Figure 5 · p. 21

E.7 POST HOC TREATMENT-DELIVERY SENSITIVITIES

  • Post hoc OpenML delivery subsets produced mixed interval scores, and each content-bearing paired comparison contained 26 states.For matched and within-shuffled cards both nonempty, interval scores were .46112 and .53691, respectively; selected-subset estimates were unstable.Source: §E.7 · p. 21, Table 23 · p. 21

E.8 DONOR RESOLUTION AND PAIRED CASES

Paired cases expose variable donor resolution, while follow-up controls test signal sensitivity across direct-text notes, component swaps, and known mechanisms. Direct-text OpenML matched gains remained uncertain, whereas researcher-authored true mechanisms improved forecasts over description.

E.8 DONOR RESOLUTION AND PAIRED CASES

  • Across 36 Tox21 pairs, median absolute realized ROC AUC-change difference was .01362; across 72 OpenML pairs, skill-change difference was .01504.Effects had the same sign in 27 Tox21 pairs and 54 OpenML pairs; only 13 OpenML elicited pairs had evidence-backed content on both sides, and one had numeric claims on both sides.Source: §E.8 · p. 21
  • In the illustrative Tox21 cell, description, matched, and donor forecasts were −.003, −.013, and +.001 against realized changes of −.02018 and −.00969.The focal card predicted −.010 and the donor card +.012; the matched explanation was more accurate in this selected cell.Source: §E.8 · p. 22
  • The sole OpenML pair with numeric claims had overlapping predicted ranges of [+.005, +.020] and [+.005, +.030], while realized changes were −.00082 and +.00358.The focal state’s description, matched, and donor forecasts were +.0065, +.0120, and +.0125.Source: §E.8 · p. 22

E.9 INTEGRITY AND USAGE

  • Elicited extraction integrity was 87/144 (60.42 percent), and all six required pooled and task-kind strata failed the .95 gate.Spontaneous extraction integrity was 118/144 (81.94 percent); 83 exhausted calls remained as ITT fallbacks.Source: §E.9 · p. 22

F KNOWN-SIGNAL CALIBRATION

  • Exact signal beat both description and wrong-seed signal in all 144 states, with one-sided task-bootstrap lower bounds of 1.0.The frozen calibration returned assay sensitive.Source: F KNOWN-SIGNAL CALIBRATION · p. 22

G.1 TOX21 NUMERICAL AND MECHANISM COMPONENTS

  • Target numbers alone reduced drift by .00181 (95 percent endpoint-bootstrap interval [.00005, .00383]); at fixed focal numbers, donor-minus-matched mechanism point MAE was −.00047 (interval [−.00228, .00104]).The complete card’s drift reduction was .00108 (interval [−.00053, .00283]); the reverse donor-number contrast was .00010 (interval [−.00183, .00181]).Source: §G.1 · p. 23

G.2 DIRECT-TEXT OPENML FORECASTS

  • Matched versus description task-balanced scaled point-MAE gain was −.09059 (95 percent task-bootstrap interval [−.26870, .04033]), with the interval spanning zero.All 144 notes were delivered directly as raw text, and both matched and donor gain intervals spanned zero.Source: §G.2 · p. 23

G.3 KNOWN-MECHANISM POSITIVE CONTROL

  • The true mechanism reduced point MAE by 2.596 points versus description (95 percent interval [2.314, 2.880]).The intervention raised held-out accuracy by 11.440 points in nonlinear tasks and changed it by −1.055 points in paired linear tasks; mechanism-conditioned forecasts moved toward these outcomes while retaining visible uncertainty.Source: §G.3 · p. 24

Related papers

Loading 2610.00314v1…