PaperlayerView exact revision on arXiv

Source-linked AI summary

MLE-World: Predicting Machine Learning Experiment Outcomes with a Learned World Model

Brian Ko, Han Wang, Hejia Geng, Li Erran Li, Sherry Yang

arXiv:2610.37175v1cs.LGcs.AI

TL;DR

MLE agents’ costly model training and evaluation limit how many candidate programs they can explore. MLE-World predicts program outcomes from code and task context, then uses those predictions and confidence scores to guide search and selective execution. Across 17 MLE-bench Lite tasks, it matches held out terminal outputs more often than prompted baselines, while guided search improves observed performance with less real execution.

  • Problem

    Training and evaluating each candidate program makes feedback costly and limits how many candidates MLE agents can explore.

    Source: §1 · p. 2

  • Method

    MLE-World is finetuned on execution logs to predict terminal outputs from program code and task context, using confidence to guide candidate selection and real execution.

    Source: §1 · pp. 2–3, §4 · p. 4

  • Results

    77.9% of held out programs’ terminal outputs match MLE-World’s predictions, versus 41.1% for the best prompted baseline, across 17 MLE-bench Lite tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    All five world-model-guided policies have higher observed mean percentiles than the agent without a world model, while confidence gating reduces real execution time by 30–40% and broadly maintains performance.

    Source: §7 · p. 10

    The main evaluation is in distribution on 17 tasks seen during finetuning; generalizable world models remain future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

AI research agents can propose, execute, and improve programs that solve machine learning tasks. However, executing each candidate program requires training and evaluating ML models, limiting how many candidates an agent can explore at each step. We introduce MLE-World, a world model finetuned from Qwen3.5-4B that predicts a program's terminal output (error traces, training logs, and validation scores) from its code and data description without executing it, together with a confidence score estimating the prediction's correctness. Because a prediction costs one inference call rather than a training run, MLE-World makes test time scaling (TTS) affordable for these agents. We build five TTS policies, guided by MLE-World, that compare several candidates per step by their predicted outputs: Sequential revision, Best-of-N, Pairwise judging, Fusion, and a UCB style rule that also adds an exploration bonus from this confidence score. Since predictions can be wrong, we run a candidate for real only when its confidence score is low. On 17 MLE-bench Lite tasks, MLE-World matches the real terminal output on 77.9% of held out programs, versus 41.1% for much larger prompted models. MLE-World guided TTS achieves up to 22.1% higher observed mean performance than the agent without a world model, with less real execution. Project webpage: https://mle-world.github.io/.

1 INTRODUCTION

MLE agents’ costly training and evaluation limit how many candidate programs they can explore, while existing pruning and ranking approaches do not provide terminal output for debugging and revision. MLE-World predicts that feedback from code and task data, enabling five prediction-guided search policies and confidence-based selective execution.

1 INTRODUCTION

  • Prior pruning and ranking methods can reduce total execution cost but leave the cost per execution unchanged and do not provide terminal output for debugging or revision.Each MLE step can require training and evaluating a model, limiting how many ideas agents can test.Source: §1 · p. 2
  • MLE-World predicts error traces, training logs, and validation scores from a candidate’s code and task data description, supporting both debugging and candidate selection.It is finetuned from Qwen3.5-4B on logged pairs of code and output.Source: §1 · p. 2
  • The five policies—Best-of-N, Sequential revision, Pairwise judging, Fusion, and UCB—compare candidate programs using predicted feedback and execute only the selected candidate.The UCB policy adds an uncertainty bonus to predicted quality; the confidence gate can work with any of the five policies.Source: §1 · p. 3
  • 77.9% of held out programs’ terminal outputs match MLE-World’s predictions, versus 41.1% for the best prompted baseline, across 17 MLE-bench Lite tasks.The paper also reports up to 22.1% higher observed mean performance for MLE-World-guided search than for the agent without a world model, with reduced real execution.Source: §1 · p. 3
  • A confidence gate runs a candidate for real only when MLE-World’s prediction has low confidence, reducing real execution time while broadly maintaining performance.The confidence score also supports exploration through the UCB policy’s uncertainty bonus.Source: §1 · p. 3

2 RELATED WORK

Prior MLE agents reduce execution cost through pruning, performance prediction, or candidate ranking, while language-model world models predict outcomes across several agent domains. MLE-World instead predicts a complete program’s terminal output to support both candidate search and revision.

2 RELATED WORK

  • MLE-World predicts complete program terminal output—including error traces, training logs, and validation scores—whereas ForeAgent uses scores and preferences without error traces for debugging.Confidence guides when MLE-World’s selected candidate needs real execution.Source: §2 · p. 3
  • Unlike DSWorld, which simulates individual data-science workflow steps and uses operation cost to decide when to simulate, MLE-World predicts a program’s terminal output and uses confidence to decide when real execution is needed.Its confidence-based gate targets unreliable feedback rather than operation cost alone.Source: §2 · p. 3
  • Sampling, tree search, revision, synthesis, and longer interaction require feedback, but slow training and evaluation constrain this exploration in MLE.MLE-World predicts terminal output to guide candidate selection, revision, and synthesis, with confidence determining when to obtain real feedback.Source: §2 · p. 3

3 BACKGROUND

The background describes AIDE’s program-search loop and places MLE-World within language-model world models. Because each candidate program is self-contained, MLE-World predicts its execution outcome from the current program and context rather than requiring interaction history.

3.1 AGENT SCAFFOLD

  • AIDE searches a program tree using Draft, Debug, and Improve operators, and each node incurs one execution to obtain logs, errors, and a validation score.The best-scoring valid node is submitted for hidden-test grading.Source: §3.1 · p. 4

3.2 WORLD MODELS FOR LLM AGENTS

  • General LLM world models condition on observation–action history, but MLE-World uses the current self-contained program and execution context to predict terminal output.The program is treated as the current observation and its execution as the action; the predicted terminal output is the next observation.Source: §3.2 · p. 4

4 MLE-WORLD

MLE-World takes program and execution context as input and predicts a structured record that supports search decisions. It is trained on logged executions, with confidence scores used for candidate selection and for deciding when to run a candidate for real.

4 MLE-WORLD

  • The predicted record contains the terminal output plus exception type, validation metric, and submission flag, decoded in that order.Exception labels are decoded first, and the model is finetuned on logged input-output pairs using supervised fine-tuning.Source: §4 · p. 4
  • MLE-World’s input includes the candidate program, installed packages and versions, a task data overview, and the execution time limit.The data overview summarizes files, CSV dimensions, columns, and example values.Source: §4 · p. 4
  • Adding explicit exception-type and validation-score fields raises terminal-output agreement from 59.4% to 62.8% and metric recall from 84.3% to 89.5%.These fields let the system represent crashes and validation scores directly rather than relying on free-text inference or a separate score extractor.Source: §4 · p. 5
  • The exception-class token probabilities define an outcome distribution, with uncertainty and clean-run probability derived from that distribution.The first predicted exception class also routes search toward Debug or Improve.Source: §4 · p. 5
  • As confidence in the most likely outcome rises, each predicted field matches the real run more often; UCB uses clean-run probability, while the gate uses the most-likely-outcome confidence.At search time, predictions also guide candidate selection and the Debug and Improve operators.Source: §4 · pp. 4–5
  • Training uses execution pairs from AIDE runs with five agent models, retaining failed episodes so the model learns outcomes for buggy candidates.The dataset spans the 17 MLE-bench tasks.Source: §4 · p. 5

5 SEARCH GUIDED BY MLE-WORLD

MLE-World-guided search uses predicted terminal outputs to compare candidates and execute a selected program, with five policies and a confidence gate. UCB uses uncertainty to encourage exploration, while the gate requests real feedback for low-confidence selections, missing confidence, or new best predicted metrics.

5.1 TEST TIME SCALING ON PREDICTED OUTPUTS

  • The five policies use predicted outputs to screen candidates and execute one: Best-of-N, Sequential revision, Pairwise judging, Fusion, and UCB.Sequential revisions use predicted feedback; Pairwise judges compare candidates, Fusion synthesizes a program, and UCB adds an exploration bonus.Source: §5.1 · p. 5
  • UCB adds outcome uncertainty to predicted quality, giving potentially underestimated candidates a chance to beat candidates with slightly better estimates.Uncertainty is an empirical exploration proxy, not a formal confidence bound, and adds no inference cost.Source: §5.1 · pp. 5–6

5.2 EXECUTION GATING BY UNCERTAINTY

  • After a policy selects a candidate, the gate requests real feedback when confidence falls below a threshold or is missing.A candidate with a new best predicted validation metric is always executed, regardless of confidence.Source: §5.2 · p. 6
  • Success and failure predictions use separate confidence thresholds because successful runs train to completion while failures usually crash within seconds.Programs predicted to time out are never run because timeout is the most costly outcome.Source: §5.2 · p. 6

6 EXPERIMENTS

Across 17 MLE-bench Lite tasks, experiments assess held-out execution prediction, world-model-guided search, and confidence gating. MLE-World-guided policies improve observed means over the unguided agent, while gating saves real execution time and broadly maintains performance.

6 EXPERIMENTS

  • The main experiments use 17 of 22 MLE-bench Lite tasks, excluding five on which the agents rarely make progress, and select each trial’s best real validation score for hidden-test grading.The prediction evaluation uses 1,700 held-out programs, 100 per task, with no exact or near-duplicate programs in training; tasks are shared by design.Source: §6 · p. 6, §6.1 · p. 7

6.1 MLE-WORLD PREDICTS EXECUTION ACCURATELY AT LOW COST

  • 77.9% terminal-output agreement on held-out programs versus 41.1% for DeepSeek, with 13–1,061× lower GPU inference time than prompted models.The evaluation uses 1,700 held-out programs; Table 1 reports prediction-field accuracy and Figure 3 plots terminal-output accuracy against GPU time.Source: Table 1 · p. 7, §6.1 · p. 7, Figure 3 · p. 7

6.2 OBSERVED SOLUTION QUALITY WITH WORLD MODEL GUIDED SEARCH

  • 52.1 mean percentile for UCB versus 45.4 for Base AIDE1; all five guided policies have higher observed means and spend less time on crashes and timeouts.The reported search comparison is ungated; Table 2 reports the search results and Figure 6 compares execution time by code outcome.Source: Table 2 · p. 8, §6.2 · p. 8, Figure 6 · p. 9
  • 20.6 versus 14.0 mean percentile for Fusion with predicted terminal output versus code alone on 150 shared states.The comparison isolates predicted terminal output by starting from the same states.Source: Table 3 · p. 8, §6.2 · p. 8
  • 45.4 versus 38.6 observed mean percentile with the Improve-stage exploration bonus versus greedy selection, varying only β.Figure 5 presents the comparison; the study focuses on Improve because most Draft and Debug candidates lack a validation metric.Source: Figure 5 · p. 8, §6.2 · p. 8

6.3 CONFIDENCE GATING: EXECUTION SAVINGS AND OBSERVED QUALITY

  • 30–40% less real execution time across the five policies with confidence gating, while broadly maintaining performance; gated Sequential has the highest observed mean percentile.Table 4 compares gated and ungated performance and execution time; Figure 10 shows trajectories.Source: Table 4 · p. 9, §6.3 · p. 9, Figure 10 · p. 9
  • Prediction matches for validation metrics, exception classes, and terminal outputs rise across confidence quintiles on 1,700 held-out programs.Figure 7 plots accuracy by pmax quintile, with task-bootstrap standard errors.Source: Figure 7 · p. 9, §6.3 · p. 9

7 LIMITATIONS AND CONCLUSION

MLE-World’s conclusion reports more accurate predictions than much larger prompted models and higher observed mean percentiles for all five guided policies, while gating reduces execution time. Its main evaluation is in-distribution, and timeouts remain its least accurately predicted outcome.

Limitations

  • The main evaluation covers new programs on 17 tasks seen during finetuning; generalizing beyond those tasks remains future work.On three tasks excluded from finetuning, the 4B model without reasoning exceeds the reported accuracy of much larger prompted models.Source: §7 · p. 10
  • Training draws on about 500,000 prior executions, including earlier trajectories from a 9B agent but no reported runs, favoring settings with accumulated logs.Real execution determines final scores, and prediction errors can misdirect search.Source: §7 · p. 10
  • Timeouts are the least accurately predicted outcome, and a missed timeout still costs a full run.The authors identify improving runtime prediction as future work.Source: §7 · p. 10

Conclusion

  • All five world model guided policies have higher observed mean percentiles than the agent without a world model, while gating reduces real execution time by 30–40%.The conclusion also reports that a 4B model predicts new programs more accurately than much larger prompted models at 13–1,061× less GPU inference time.Source: §7 · p. 10

ETHICS STATEMENT

The experiments use public Kaggle competition data distributed with MLE-bench and involve no human subjects or personal data.

Ethics Statement

  • The experiments use only public Kaggle competition data distributed with MLE-bench.The authors state that the work raises no ethical concerns.Source: ETHICS STATEMENT · p. 10
  • The study involves no human subjects or personal data.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility materials cover training, prediction, search, and evaluation procedures, and the authors say they will release the checkpoint, corpus, prompts, and code.

Reproducibility Statement

  • Appendices document the training data and configuration, deduplication and holdout, decoding, prediction prompt, and output parsing.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The agent results report seeds, budgets, and evaluation protocols; the model was trained on 8 NVIDIA H200 GPUs.The full training configuration is in Appendix B.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors say they will release the MLE-World checkpoint, execution corpus with its held-out split, prompts, and agent and evaluation code.Source: REPRODUCIBILITY STATEMENT · p. 10

LLM USAGE

The paper reports LLM use in program generation, judging, and writing, and describes the training corpus, model configuration, and predicted exception classes.

LLM USAGE

  • Qwen3.5 models generate agent programs, Qwen3-Coder-Next judges outputs, and an LLM coding agent developed the judge prompt under Appendix I’s protocol.LLM assistants also helped write experiment and analysis code and edit the text; the authors remain responsible for all content.Source: LLM USAGE · p. 10

A TRAINING DATA

  • The 497,056-program training set contains 52.3% crashes and 7.0% timeouts; every holdout task is also a training task, though held-out programs are excluded.The evaluation keeps 17 tasks where agents often produce working programs; four excluded image tasks have only 2.3–11.6% clean submissions, versus 38.6% on kept tasks.Source: Table 5 · p. 13, A TRAINING DATA · p. 13

B TRAINING CONFIGURATION

  • All Qwen3.5-4B parameters are trained for five epochs on 8 H200 GPUs, and the final checkpoint is evaluated.Inputs include the program and package versions, task data overview, and execution time limit; the final checkpoint is step 6,685.Source: B TRAINING CONFIGURATION · p. 14, Table 6 · p. 15

C EXCEPTION CLASSES

  • MLE-World predicts 82 exception classes, each added as a single tokenizer token; the set covers every exception class in the training corpus.The first decoded token gives the distribution used to derive pmax and pclean.Source: C EXCEPTION CLASSES · p. 15

D TARGET ABLATION

This section evaluates ablations and settings for outcome prediction, prediction-guided candidate selection, and confidence-based execution gating. It covers terminal-output matching and evaluation configuration, UCB selection and exploration sensitivity, and the runtime–quality trade-offs of gating.

D TARGET ABLATION

  • 90.0% metric precision from a metric-only target came with 77.0% metric coverage, versus 89.5% for the full target, which also supplies error traces for debugging.The full target gives up 1.0 point of precision for 12.5 points of recall; trimming logs to the first and last 2,500 characters showed little observed difference from untrimmed logs, though one run per condition does not establish equivalence.Source: D TARGET ABLATION · p. 15

E DEDUPLICATION AND HOLDOUT CONSTRUCTION

  • Deduplication reduced 757,186 executions to 602,741; filtering and task-balanced holdout sampling produced 498,756 executions, including 100 held-out programs per task and 497,056 for training.No exact or near duplicates were detected between holdout and training using the stated procedure, though MinHash/LSH can miss similar pairs and does not rule out semantic or cross-task overlap; held-out programs come from tasks represented in training.Source: E DEDUPLICATION AND HOLDOUT CONSTRUCTION · pp. 16–17

D TARGET ABLATION

  • 62.8% terminal-output agreement, 75.3% exception-class agreement, and 69.1% crash-class agreement followed adding exception class and validation metric, up from 59.4%, 71.7%, and 64.8%.The explicit fields also improved the generated log itself, rather than merely providing separate outputs.Source: D TARGET ABLATION · p. 15, Table 8 · p. 17

F INFERENCE AND JUDGE SETTINGS

  • MLE-World and Coder decode greedily, while DeepSeek samples at temperature 1.0 and top-p 0.95 with reasoning enabled; the Coder judge sees only predicted and observed outputs.Coder is also a baseline and is the least accurate model on the judged fields, but the authors do not rule out judge bias toward its own outputs.Source: F INFERENCE AND JUDGE SETTINGS · pp. 17–18
  • Corpus baselines copy training or parent records rather than using a model, and Table 10 reports that they trail MLE-World on every field and stage.The baselines include task majority, nearest program, parent record, and random record.Source: Table 10 · p. 19, F INFERENCE AND JUDGE SETTINGS · p. 19

G OUTPUT PARSING

  • Both prompted models return all three scored fields for over 91% of programs, so parsing failures cannot explain their 28–60 percentage-point accuracy gap in Table 1.DeepSeek’s output is longer because it reasons, whereas the other two models run with reasoning off.Source: Table 11 · p. 20

H ACCURACY BREAKDOWN

  • MLE-World leads both prompted baselines on every reported stage, outcome, and crash-type slice and field except IndexError crash-class accuracy; timeouts are its hardest case.The IndexError exception-class slice contains 26 programs.Source: H ACCURACY BREAKDOWN · p. 20, Table 12 · p. 21

I JUDGE PROMPT AND VALIDATION

  • The judge prompt agreed with development labels on 491 of 496 decisions, then was tested on a separately frozen, human-labeled set of 120 cases with no shared programs or parent-child pairs.The prompt was selected before test inputs were viewed, and test labels were frozen before scoring; Table 13 reports agreement over three judge requests per case.Source: I JUDGE PROMPT AND VALIDATION · p. 22, Table 13 · p. 22
  • The judge compares fatal outcomes by actionable debugging cause and checks corresponding printed dataset sizes before crash matching; clean headline results match only when corresponding metrics are within 15%.A dataset-size ratio above 2 rejects a match, while fatal comparisons allow a paired pre-crash metric gap up to 50%; clean or recovered outputs have debug_match=null. The judge distinguishes diagnoses by implicated objects or operations rather than superficial wording.Source: I JUDGE PROMPT AND VALIDATION · pp. 25–26

J TEST TIME SCALING ALGORITHMS

  • The five selection policies use predicted outputs to choose among candidates through sequential refinement, predicted-metric greedy selection, pairwise elimination, fusion, or metric UCB with a confidence-based exploration bonus.UCB uses the same ordering as metric-greedy selection for predicted quality; missing confidence disables the bonus for the pool.Source: J TEST TIME SCALING ALGORITHMS · pp. 28–29

K UCB SELECTION ABLATIONS

  • The ablation compares matched arms across 17 tasks and three seeds, using eight-hour runs and requesting a real check for every selection subject to timeout and budget constraints.Mean percentile weights tasks equally after averaging seeds; invalid or missing submissions score zero.Source: K UCB SELECTION ABLATIONS · p. 29

K.1 IMPROVE STAGE BONUS WITH A SHARED GREEDY CONTROL

  • 6.80 mean percentile points: Improve-stage UCB outperforms metric-greedy selection, with 1,249 versus 994 Improve decisions under a matched time budget.Only the exploration bonus changes between these arms; their metric scaling, usability, reviewer exclusions, and fallback are matched.Source: Table 14 · p. 29, Figure 8 · p. 29, §K.1 · p. 30

K.2 DEBUG STAGE SELECTOR COMPARISON

  • 6.49 percentile points: Debug-stage UCB has a higher observed mean than pclean-greedy selection, but the difference is not established; both arms yield 48/51 valid submissions.This comparison changes both reward and eligibility, and does not test a failed-pool judge.Source: Table 14 · p. 29, Figure 8 · p. 29, §K.2 · p. 30

L UCB EXPLORATION WEIGHT SENSITIVITY

  • The exploration-weight study tests β values 0, 0.25, 0.5, 1, and 2 at three-hour checkpoints, then reports completed eight-hour runs for β = 0.5 and 1 with and without the S50/F83 confidence gate.These controls precede the 12-hour all-stage UCB results, and comparisons across experiment waves are descriptive.Source: L UCB EXPLORATION WEIGHT SENSITIVITY · p. 30, Table 15 · p. 30

M GATE THRESHOLD SELECTION

  • Held-out threshold selection evaluated separate success and failure gates, trading prediction error against the fraction of runtime skipped.For successes, γs = 0.90 yielded 15.5% error and 67.7% of runtime skipped, while γs = 0.50 yielded 17.8% error and 99.6% skipped. For failures, γf = 0.83 yielded 15.0% error and 55.9% skipped, while γf = 0.56 yielded 19.9% error and 87.7% skipped. Predicted timeouts were never run; running them would occupy the executor for the full time limit.Source: M GATE THRESHOLD SELECTION · p. 31, Figure 9 · p. 32
  • A three-hour development sweep selected γs = 0.50 for every policy; γf = 0.56 for Sequential, Pairwise, and BoN, and 0.83 for Fusion and UCB.These selected thresholds were frozen for the earlier eight-hour evaluation; the twelve-hour comparison also reports an S50/F56 UCB extension.Source: M GATE THRESHOLD SELECTION · p. 31, Table 16 · p. 31

N BON RANDOM EXECUTION CONTROL

  • 19% less real execution time—4.49 versus 5.54 hours per trial—was used by confidence gating, with no established quality difference (−0.15 ± 2.32 percentile points).The comparison averaged 51 eight-hour trials per gate across 17 tasks.Source: N BON RANDOM EXECUTION CONTROL · p. 32, Table 17 · p. 33

O AGENT EXECUTION STATISTICS

Execution statistics show that confidence gating reduces real runs, while guided search shifts execution toward cleaner outcomes and Improve-stage work. In fixed-state tests, screening and Fusion improve immediate validity and percentile, but the local comparisons do not establish that real feedback adds further gains or predict full-search outcomes.

O AGENT EXECUTION STATISTICS

  • Guidance shifts real execution from Debug toward Improve; gated policies spend 40–52% of execution in Improve, versus Base AIDE’s 39%.Clean Improve runs total 1.19 hours for Base AIDE and 1.19–2.25 hours across the five policies.Source: O AGENT EXECUTION STATISTICS · p. 34
  • Fusion’s 2.10-point overall lead over UCB decomposes into 0.12 points on shared valid pairs and 1.98 points on pairs where only one submits validly.Fusion publishes verified submissions on five of UCB’s six corresponding failed pairs, but the records do not isolate a causal effect of UCB’s uncertainty bonus.Source: O AGENT EXECUTION STATISTICS · p. 34
  • The confidence gate removes 48–73% of remaining real runs; clean-run share rises for four of five policies, and all five lose less time to timeouts.The table covers 51 trials per row across 17 tasks and three seeds.Source: O AGENT EXECUTION STATISTICS · p. 33, Table 19 · p. 35

P DIAGNOSTICS ON FIXED STATES

  • Screening raises chosen-program validity from 26.1% to 41.9%, and predicted-feedback Fusion raises it from 36.7% to 49.2%; test percentile rises by 5.51 and 6.55 points, respectively.The percentile gains arise mostly through higher validity; ranking screened candidates adds only 0.68 points after screening.Source: P DIAGNOSTICS ON FIXED STATES · p. 36
  • Fusion’s mean percentile is 20.6 on shared fixed states, versus 17.3 for metric-greedy selection, 15.4 for Pairwise, and 15.3 for Sequential.The comparisons support Fusion’s advantage over Pairwise and Sequential; its higher mean than metric selection is descriptive.Source: P DIAGNOSTICS ON FIXED STATES · p. 37
  • The fixed-state diagnostics are not full-search comparisons: they measure one local output with different model-call and execution costs, not final submissions after evolving eight-hour searches.The real-execution BoN reference reports 44.96 mean leaderboard percentile at 11.90 hours of execution and cleanup per trial, but its archived-cohort comparison is descriptive.Source: O AGENT EXECUTION STATISTICS · p. 33, Table 18 · p. 34, P DIAGNOSTICS ON FIXED STATES · pp. 36–37

Q CHOICE OF PRIMARY AGENT METRIC

Medal thresholds make gold rates sensitive to a few tasks and submissions, while percentile distinguishes performance across more tasks. Yet percentile can swing sharply when leaderboard scores cluster, so the paper uses mean percentile as its primary metric and reports medal rates for comparison.

Q CHOICE OF PRIMARY AGENT METRIC

  • A single seed shifts a policy’s gold rate by 2.0 points, and on aerial-cactus gold requires an exact AUC of 1.0 despite a 0.99996 score ranking at the 87.7th percentile.With three seeds per task, threshold-based medals can hinge on one result.Source: Q CHOICE OF PRIMARY AGENT METRIC · p. 38
  • Fusion’s gold rate is 5.9% versus Base AIDE’s 11.8%, yet Fusion has 96.1% valid submissions versus 78.4% and a 10.4-point higher mean percentile.Gold rate can therefore reverse the policy ranking suggested by mean percentile.Source: Q CHOICE OF PRIMARY AGENT METRIC · p. 38, Table 22 · p. 40
  • On English text normalization, accuracy 0.964 ranks at the 6th percentile, while 0.990 ranks at the 47th.Averaging percentile over 17 equally weighted tasks limits any one leaderboard’s influence; the paper reports medal rates for comparison.Source: Q CHOICE OF PRIMARY AGENT METRIC · p. 38
  • More than half of all golds come from one task, while 12 tasks yield no gold and nine yield no medal of any kind.Percentile still separates policies on tasks without medals.Source: Q CHOICE OF PRIMARY AGENT METRIC · p. 38, Table 21 · p. 39

R EXECUTION PREDICTION ON TASKS EXCLUDED FROM FINETUNING

The paper explores whether MLE-World’s execution predictions transfer to three tasks excluded from finetuning. The results suggest transfer, but the evidence mainly concerns recognizing execution failures, leaving successful training outcomes and timeouts less established.

R EXECUTION PREDICTION ON TASKS EXCLUDED FROM FINETUNING

  • The transfer evaluation uses 300 programs from three tasks excluded from finetuning, balanced across Draft, Improve, and Debug stages.An audit found neither these task identities nor exact program matches in the finetuning examples after whitespace normalization.Source: R EXECUTION PREDICTION ON TASKS… · p. 40, Table 23 · p. 40
  • Shared operations such as image loading, tensor transformations, and pretrained-model adaptation are proposed as plausible routes for transfer, not measured causal mechanisms.Task-specific geometry, labels, and metrics may involve less familiar behavior.Source: R EXECUTION PREDICTION ON TASKS… · p. 40, Table 24 · p. 41
  • The results suggest transfer to excluded tasks, but failed executions dominate the sample, so evidence mainly concerns recognizing failures; successful training outcomes and timeouts remain less established.The authors call for broader prospective evaluation and training-family ablations to distinguish transfer routes.Source: R EXECUTION PREDICTION ON TASKS… · p. 40, Table 25 · p. 41

Related papers

Loading 2610.37175v1…