Can We Predict Before Executing Machine Learning Agents?

Jingsheng Zheng; Jintian Zhang; Yujie Luo; Yuren Mao; Yunjun Gao; Lun Du; Huajun Chen; Ningyu Zhang

Source-linked AI summary

Can We Predict Before Executing Machine Learning Agents?

Jingsheng Zheng, Jintian Zhang, Yujie Luo, Yuren Mao, Yunjun Gao, Lun Du, Huajun Chen, Ningyu Zhang

arXiv:2601.05930v2Jan 9, 2026cs.CLcs.AIcs.LGcs.MA

TL;DR

Autonomous ML agents face an execution bottleneck because evaluating hypotheses relies on expensive physical runtime checks. This paper formalizes pre-execution Data-centric Solution Preference, builds a large comparison corpus, and integrates predictive filtering into FOREAGENT. LLMs achieve 61.5% accuracy, while FOREAGENT reaches 6× acceleration and +6% performance over standard baselines.

Problem
Autonomous ML agents commonly rely on costly physical execution within a Generate-Execute-Feedback loop, creating an execution bottleneck.
Method
The paper formalizes Data-centric Solution Preference, constructs pairwise comparisons, and uses Verified Data Reports with a Predict-then-Verify loop in FOREAGENT.
Results
61.5% accuracy is achieved by DeepSeek-V3.2-Thinking, while FOREAGENT delivers 6× acceleration and +6% performance over standard baselines.
Takeaways & Limitations
Predictive reasoning can filter candidate solutions before physical execution while retaining verification through a final executed candidate.
Takeaways & Limitations
The corpus is skewed toward mainstream tasks, so reliability may vary in extremely low-resource or highly specialized scientific domains.

Abstract

from arXiv · show

Autonomous machine learning agents have revolutionized scientific discovery, yet they remain constrained by a Generate-Execute-Feedback paradigm. Previous approaches suffer from a severe Execution Bottleneck, as hypothesis evaluation relies strictly on expensive physical execution. To bypass these physical constraints, we internalize execution priors to substitute costly runtime checks with instantaneous predictive reasoning, drawing inspiration from World Models. In this work, we formalize the task of Data-centric Solution Preference and construct a comprehensive corpus of 18,438 pairwise comparisons. We demonstrate that LLMs exhibit significant predictive capabilities when primed with a Verified Data Analysis Report, achieving 61.5% accuracy and robust confidence calibration. Finally, we instantiate this framework in FOREAGENT, an agent that employs a Predict-then-Verify loop, achieving a 6x acceleration in convergence while surpassing execution-based baselines by +6%. Our code and dataset are publicly available at https://github.com/zjunlp/predict-before-execute.

1 Introduction

Autonomous ML agents are limited by costly execution-based feedback, motivating prediction of solution quality before runtime. The paper formalizes this capability and operationalizes it in FOREAGENT, reporting faster and stronger search.

Execution bottleneck: Up to 9 hours per run in MLE-Bench illustrates the latency imposed by execution-based agent feedback.Mainstream agents iteratively generate, execute, and refine code using runtime output.
From execution to inference: Implicit Execution Priors replace costly runtime checks with instantaneous predictive reasoning inspired by World Models.The approach seeks to evaluate actions internally rather than through external trials.
Task formulation: Data-centric Solution Preference asks models to predict the relative performance of two algorithmic solutions from a data analysis report without physical execution.The task evaluates whether data-grounded solutions can be ranked through reasoning alone.
Dataset: 18,438 pairwise comparisons form the corpus used to evaluate predictive solution preference.The corpus is presented as a comprehensive resource for this task.
FOREAGENT: FOREAGENT uses a Predict-then-Verify loop to decouple exploration from execution, expanding the search space by 3.2×, accelerating convergence 6×, and improving performance by +6% over the baseline.The agent applies predictive filtering before physical verification.

2 Background

Autonomous ML agents seek high-performing code but commonly depend on execution feedback, which is computationally expensive. This work frames pre-execution ranking as pairwise preference prediction over data-grounded candidate solutions.

Task objective: An autonomous ML task seeks code C* that maximizes metric M on dataset D given instruction I.The formal objective selects an optimal solution from the code space.
Existing paradigm: Current agents typically follow a Generate-Execute-Feedback paradigm for iterative solution improvement.AIDE exemplifies tree-search exploration through drafting, debugging, and execution feedback.
Corpus: The Preference Corpus aggregates 26 tasks into three domains with approximately 6,000 pairs per domain.The corpus statistics describe a balanced high-level domain distribution.
Execution bottleneck: Physical execution is the primary constraint because deep-learning training is computationally heavy and can produce timeout failures.The resulting efficiency gap motivates compressing execution into logical inference.
Implicit world modeling: Data-centric Solution Preference determines which of two candidate solutions is superior using reasoning about algorithmic logic and stochastic data.Unlike explicit state tracking, the task targets invisible coupling between algorithmic logic and data.

3 Preference Corpus Curation

The paper formalizes Data-centric Solution Preference as pairwise selection from task descriptions, data reports, and candidate solutions, then builds a large corpus from curated agent trajectories. The resulting task predicts the superior solution and confidence without physical execution.

Task Definition: Data-centric Solution Preference selects the superior solution from a task, data report, and two candidate solutions.The task also requires estimating a confidence score.
Task Definition: The formal input X contains the task, data report, candidate code pair, and system prompt.The output includes reasoning, the predicted winner, and confidence used as a gating threshold.
Trajectory Collection: The corpus begins with 1,329 valid solutions generated across 26 diverse MLE-bench tasks by AIDE and AutoMind.These candidates are complete agent-generated machine-learning workflows rather than synthetic snippets.
Corpus Construction: 18,438 pairwise comparisons remain after expert curation, deduplication, diversity controls, ambiguity filtering, and position balancing.The pipeline prunes raw trajectories to 895 high-quality instances before exhaustive pair construction.
Input Augmentation: Verified Data Analysis Reports transform raw statistics into semantic narratives to address models’ numerical and context limitations.A generated profiling script supports factual grounding before the report is used for preference prediction.

4 Main Experiments

The main experiments evaluate LLM pairwise preference accuracy against random and complexity-based baselines over 18,438 comparisons. Both tested models outperform these baselines, establishing predictive capability from static inputs.

Feasibility: The stratified pairwise accuracy results validate the feasibility of run-free preference prediction.Both evaluated models significantly outperform the two baselines.
Main Results: 61.5% Pairwise Preference Accuracy is achieved by DeepSeek-V3.2-Thinking, exceeding random guessing at 50.0% and the complexity heuristic at 50.8%.The primary metric is micro-averaged accuracy across 18,438 pairwise comparisons.
Main Results: 58.8% Pairwise Preference Accuracy is achieved by GPT-5.1, also significantly outperforming the random and complexity-based baselines.The complexity heuristic assumes that the more complex solution is better.

5 Analysis & Insights

The analyses attribute predictive performance to semantic data understanding and active reasoning, while identifying limits from task complexity, listwise scaling, and parameter-only scaling. Confidence calibration supports selective use, but global ranking remains weak.

Data Representation: 61.3% accuracy with Verbal Reports exceeds 59.0% with Numerical Stats, 56.7% with Code Only, and 50.8% with the Heuristic Baseline.The 56.8% Context Mismatch result shows that irrelevant context adds little over Code Only, supporting semantic alignment as the key factor.
Reasoning and Boundaries: 61.3% Thinking Mode accuracy exceeds 55.9% Direct Answering accuracy, remaining robust across temperatures from 0 to 1.5.Performance varies across tasks and solutions, including 66.9% for NLP and 63.9% for Easy paradigms.
Reasoning and Boundaries: 62.8% accuracy for Cross-Algo contrasts exceeds the model’s performance on finer-grained distinctions, while complex code incurs a 59.6% Complexity Tax.The analysis also reports 64.5% preference for Traditional ML within the Algo Era.
Ranking Limitations: Accuracy@1 drops from 61.3% pairwise accuracy to 31.1% in listwise ranking, while Spearman correlation remains approximately 0.23.The results indicate weak global discrimination beyond binary interactions.
Calibration: Accuracy scales with the complexity gap, and confidence correlates with accuracy, indicating calibrated predictions rather than reliance on simple heuristics.The figure analysis links these findings to semantic signals and action gating.
Scaling Laws: Performance plateaus after the initial 30B threshold and persists at the 1T scale, showing that raw parameter scaling alone is insufficient.The paper attributes future gains to reasoning-centric architectural paradigms rather than simple parameter expansion.

6 Agent Integration: FOREAGENT

FOREAGENT decouples hypothesis exploration from physical execution through a conservative Predict-then-Verify loop using an Implicit World Model. Evaluated on five MLE-bench AI4Science tasks, it improves efficiency and Beat Ratio relative to AIDE while targeting generalization to unseen tasks.

Method: FOREAGENT uses an Implicit World Model to filter hypotheses before execution, decoupling exploration from physical verification.The framework is designed to break the execution bottleneck by pruning the search space before execution.
Method: The Predict-then-Verify loop generates m = 10 candidates, applies a confidence gate c = 0.7, and physically verifies the Top-k k = 1 candidate.This conservative workflow separates high-volume generation, confidence-gated pairwise selection, and final verification execution.
Evaluation: FOREAGENT is evaluated on 5 AI4Science tasks from MLE-bench, including two unseen tasks, against AIDE under a 12-hour limit.Both agents use DeepSeek-V3.2 for coding, while implicit world modeling uses DeepSeek-V3.2-Thinking.
Results: 6× speedup lets FOREAGENT converge using 1/6 of the execution time while exploring 3.2× more nodes than the baseline.The expanded search breadth results from substituting costly execution with rapid inference.
Results: +6% average Beat Ratio improvement over AIDE accompanies the efficiency gains and robust generalization on unseen tasks.Beat Ratio is reported as the percentage of human leaderboard contestants outperformed by the agent, averaged across three independent runs per task.

7 Related Work

Related work addresses the generation-execution bottleneck through internal priors, world-model-based execution prediction, and reasoned pruning. This paper combines those directions with data-centric solution preference grounded in explicit dataset rationales.

ML Agents: LLM-based ML agents commonly rely on generation-execution loops, motivating methods that use internal priors to prune redundant steps.These approaches seek to move from brute-force search toward reasoned planning.
World Models: World-model research adapts environmental simulation to code by predicting execution outcomes or bypassing physical runs.Prior work also evaluates logic consistency directly in reasoning benchmarks.
Positioning: This work integrates predictive execution capability with Data-Centric Solution Preference, anchoring evaluations in explicit dataset rationales rather than heuristics.The stated goal is reliability in stochastic data-analysis settings.

8 Conclusion

The paper presents physical-execution compression into logical inference as feasible for complex machine-learning tasks. It characterizes LLMs as calibrated, reasoning-driven critics that gate actions and prune search spaces.

Conclusion: The framework validates compressing physical execution into logical inference for complex machine-learning tasks.Its analysis frames this as a way to address the execution bottleneck.
Conclusion: LLMs function as calibrated, reasoning-driven critics that use semantic verbalization to gate actions and prune search spaces.The conclusion presents decoupling reasoning from runtime as a blueprint for bypassing the execution bottleneck.

Limitations

The paper identifies limitations in corpus coverage and agent implementation. These include skew toward mainstream tasks, metadata-based reports for unstructured domains, and limited exploration of inference-time design choices.

Corpus Coverage: The 18,438-pair corpus is imbalanced, with Classification and Regression dominating while niche scientific tasks have fewer samples.Reliability may vary in extremely low-resource or highly specialized domains, and evaluation may not fully reflect long-tail challenges.
Corpus Coverage: Verified Data Reports rely on metadata for unstructured CV and NLP domains, leaving multimodal semantic profiling for future work.The limitation concerns the current handling of unstructured data analysis.
Agent Implementation: FOREAGENT alternates strictly between singular prediction and execution, leaving advanced inference-time strategies, architectures, and hyperparameter configurations unexplored.The authors state that this implementation has not been pushed to its optimal limit.
Agent Implementation: The reported performance is therefore presented as a lower bound of the framework’s capability.The claim follows from the limited exploration of possible inference-time strategies in the current instantiation.

B.1 Task Metadata and Scale

The corpus covers 26 diverse tasks and heterogeneous machine-learning solutions, including seen and unseen scientific problems. Its preference pairs span complete exploration trajectories and logically imperfect intermediate states.

Solution Diversity: The solution space includes traditional statistical methods, gradient-boosting trees, neural networks, CNNs, and Transformers.The corpus is described as a heterogeneous solution manifold across major machine-learning paradigms.
Task Coverage: 26 tasks span diverse domains, with evaluation covering both seen and unseen AI4Science problems.Unseen tasks include Aerial Cactus and Histopathologic Cancer Detection.
Trajectory Sampling: Pairs are drawn across the entire valid search path rather than only from final best solutions.This design tests preference prediction among messy, unfinalized intermediate code states.
Evaluation Structure: The benchmark reports granular performance analyses across individual tasks, domains, agent architectures, and task paradigms.The breakdown separates Ranking, Matching, Segmentation, and Extraction within the broader task categories.

C.2 Analysis of Pair Source and Trajectory Variance

Trajectory-source analyses indicate that predictive accuracy is not inflated by within-trajectory pairs, while FOREAGENT improves local decision reliability over execution-only feedback. The selected k = 1 setting balances stability and improvement, whereas larger k increases exposure to noisy validation signals.

Pair-Source Robustness: 60.4% within-trajectory accuracy for DeepSeek-V3.2 is slightly below 61.7% cross-trajectory accuracy, indicating no pair-source inflation.GPT-5.1 performance is statistically identical across the two subsets.
Trajectory Interpretation: Complete runnable pipelines appear even at early trajectory stages, so distinguishing solutions requires algorithmic reasoning rather than syntax-error detection.Later steps typically refine methodology, such as switching algorithms.
Decision Reliability: 53.49% Test Improve Rate for FOREAGENT exceeds the execution-only baseline’s 30.39% in noisy local iterations.The World Model filters candidates with high validation scores but flawed logic before execution.
Decision Reliability: 75.23% Val vs Test Agreement remains consistent with the reported global 72% theoretical ceiling.The local-improvement agreement is accompanied by a higher actual success rate for FOREAGENT.
Top-k Selection: k = 1 balances 75.23% Val Test Agreement with 53.49% Test Improve Rate, while k = 2 raises average node count to 157.87 and destabilizes exploration.The study identifies k = 1 as optimal but leaves broader hyperparameter tuning for future work.

D.1 Case I: Overcoming Complexity Bias (Reasoning Analysis)

The case studies show the World Model using data and domain constraints to reject complex-looking but mismatched architectures. Its reasoning combines generated data reports with candidate code to prefer solutions aligned with dataset scale, resolution, and task structure.

Complexity Bias: Small data and skewed targets make the complex DNN vulnerable to overfitting, so the World Model prioritizes the LightGBM ensemble.The Google Quest task has approximately 5.5k samples and strong Question-Group structure.
Domain Fit: The World Model rejects ViT-B/16 because resizing 101 × 101 seismic images to 224 × 224 introduces interpolation noise and destroys fine-grained details.It instead selects U-Net to preserve native spatial resolution and avoid severe domain mismatch.
Grounded Reasoning: A Verified Data Analysis Report converts execution logs and raw statistics into semantic narratives that the reasoning engine can use as grounded context.The result-prediction prompt integrates the task description, report, and candidate solution code.
Prompt Workflow: The prompt workflow includes data-analysis code generation, report generation, result prediction, and auxiliary complexity scoring.Complexity scoring evaluates code engineering, model architecture, and data-pipeline dimensions.
Prediction Task: The preference task asks the model to determine which of two algorithmic solutions is superior using reasoning before physical execution.The analysis frames this as a pairwise preference problem over data-grounded solution quality.