PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Automatic Research Agents via World Models

Xiyuan Yang, Sheikh Sarwar, Jingru Cheng, Zhan Shi, Duanshun Li, Huiyuan Chen, Haiyang Zhang, Xing Fan, Chenlei Guo, Jingrui He, Zhenyu Liao

arXiv:2608.12564v3cs.LG

TL;DR

Scaling RL for AutoResearch is limited by expensive environment execution, which cannot share compute like batched agent generation. WMRL replaces execution with a world model, corrects its biased and noisy rewards using anchored real execution, and cuts training compute by three to four times while matching or exceeding full real-execution RL.

  • Problem

    AutoResearch RL requires massive trajectories, but batched generation scales differently from exclusive-sandbox execution, making environment execution the training bottleneck.

  • Method

    WMRL replaces real execution with a world model and uses Online Debiasing plus Inverse-Variance Denoising with a small anchored stream of real execution.

  • Results

    WMRL accelerates overall training by 3–4×, while its corrected training matches or exceeds real-environment training across AutoResearch tasks and scales.

  • Takeaways & Limitations

    WMRL provides a way to scale RL when execution, rather than generation, is the bottleneck, with transfer demonstrated on embodied VLA post-training.

  • Takeaways & Limitations

    WMRL still requires a small stream of real execution to estimate and correct world-model bias and noise.

Abstract

from arXiv · show

Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory (agent generation and environment execution) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow. To resolve this tension, we propose World Model RL (WMRL), which replaces environment execution with a world model to remove this bottleneck. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise. Therefore, we further equip WMRL with two mitigations, Online Debiasing and Inverse-Variance Denoising, which offset the bias and suppress the noise respectively. Theoretically, we prove that both mitigations of WMRL strictly improve the convergence guarantee. Empirically, WMRL accelerates training by 3-4x on various tasks at different agent scales, while exceeding the performance of standard RL baselines. Moreover, our post-trained 4B and 9B agents outperform much larger open-weight agents of 48B and 120B on held-out benchmarks. Beyond AutoResearch, WMRL also transfers to post-training embodied VLA policies, which demonstrates the generalizability of our method.

1. Introduction

AutoResearch agents use execution outcomes to support reinforcement learning, but environment execution scales poorly compared with batched generation. WMRL replaces this bottleneck with a world model and corrects its reward errors using anchored real execution.

  • 1. Introduction: Generation shares compute through batching, whereas every candidate solution requires an isolated sandbox with real GPU execution, causing execution to dominate at scale.This asymmetry motivates replacing execution with a scalable learned signal.
  • 1. Introduction: WMRL replaces expensive environment execution with a world model, removing the critical scaling bottleneck and accelerating training by 3–4×.The world model produces execution signals through a few forward passes rather than real sandboxed execution.
  • 1. Introduction: Online Debiasing and Inverse-Variance Denoising counteract world-model reward bias and noise, so accelerated training matches or exceeds real-environment training.The two mechanisms respectively recalibrate biased rewards and reduce update variance.
  • 1. Introduction: An imperfect world model adds O(B^2) and O(σ^2) convergence errors, and the proposed mechanisms strictly improve the resulting convergence guarantee.The bias and noise terms arise from deviations between predicted and real execution rewards.
  • 1. Introduction: Experiments evaluate efficiency and performance across AutoResearch tasks and agent scales, then extend WMRL to embodied VLA post-training.The introduction identifies VLA transfer as evidence of generalizability.

2. Related Work

Related work places WMRL at the intersection of AutoResearch agents, learned reward signals, and world models used as execution surrogates. Prior work also documents errors in learned proxies, motivating explicit correction mechanisms.

  • 2. Related Work: AutoResearch research spans autonomous experimentation, scientific discovery, machine-learning engineering, agentic data science, and recursive self-improvement.Benchmarks include real Kaggle-style competitions such as MLE-Bench.
  • 2. Related Work: Learned signals already replace expensive ground truth in RLHF, LLM judging, model-based RL, physical-world foundation models, and software-agent execution surrogates.This literature also documents errors in such proxies, providing context for WMRL’s reward corrections.

3. Method

WMRL replaces real execution with a language-model world model while retaining a small anchored stream of real rewards. It calibrates bias online and fuses reward streams to reduce variance, with safeguards for unreliable predictions and transfer beyond AutoResearch.

  • 3. Method: WMRL uses a language model to predict execution outcomes, letting RL train on scalable surrogate rewards instead of exclusive-GPU real executions.The world model receives task context and the agent solution, then predicts the execution outcome.
  • 3.3. Anchor Signal for Error Correction: World-model rewards decompose into real rewards plus bounded bias and zero-mean noise, which introduce O(B^2) and O(σ^2) convergence errors.Theoretical analysis compares these additional terms with the standard real-execution convergence bound.
  • 3.3. Anchor Signal for Error Correction: A small anchor stream grades groups with both real execution and the world model, supplying score pairs for reward correction.About 10% of groups serve as anchors in practice.
  • 3.3. Anchor Signal for Error Correction: Online Debiasing fits a monotone mapping to anchor score pairs and refits it during training, recalibrating world-model rewards before advantages are formed.The mapping is obtained through isotonic regression and tracks reward drift over training.
  • 3.3. Anchor Signal for Error Correction: Inverse-Variance Denoising combines true-score anchor gradients with abundant calibrated world-model gradients, weighting streams to minimize variance.The method estimates the variance ratio from residuals between calibrated and true anchor scores.
  • 3.4. Discussion: When prediction residuals worsen, WMRL increases anchor weighting and can degrade toward standard GRPO for outcomes dominated by unobservable randomness.This provides an adaptive fallback rather than relying on a fixed mixing ratio.
  • 3.4. Discussion: The design transfers to settings where rewards are expensive but predictable from agent artifacts and a small ground-truth stream remains available, including embodied VLA post-training.The paper validates this broader setting on VLA tasks.

4. Theoretical Analysis

The analysis shows that imperfect world-model rewards add separate bias and variance errors to RL convergence, while WMRL’s two corrections reduce these penalties and approach real-environment training.

  • 4.1. Analysis Setup: The analysis measures progress after T ascent steps by the expected gap to the optimal score under smoothness, gradient domination, and bounded log-likelihood gradients.These conditions are stated as Assumption 2 for the convergence analysis.
  • 4.2. The Cost of World Model Rewards: Theorem 3 attributes world-model error to bias bounded by B and zero-mean noise with standard deviation σ, contributing O(B^2) and O(σ^2) convergence terms.The gradient estimator transmits bias through its mean and noise through its variance.
  • 4.2. The Cost of World Model Rewards: World-model bias creates a permanent convergence floor, whereas world-model noise inflates the final error through the variance term.The bias term is independent of training duration and step size, while the variance term grows with VWM.
  • 4.3. The Effect of Our Mitigation: Online Debiasing uses anchor-based score pairs to shrink residual bias, while Inverse-Variance Denoising fuses gradient streams to reduce noise.Real execution grades only a small fraction of groups, supplying the anchor signal used by both corrections.
  • 4.3. The Effect of Our Mitigation: WMRL’s convergence bound makes each remaining error term strictly smaller than the corresponding world-model-only term, with the bias term vanishing as training grows.The leading terms coincide; the bias contracts by 1 + T/T0, while the variance contracts through inverse-variance fusion.

5. Experiments

Experiments show that WMRL reduces training compute while improving AutoResearch results, transfers to VLA post-training, and benefits from both correction mechanisms.

  • 5.1. Experimental Setup: Experiments cover AutoResearch tasks at 4B and 9B scales, held-out benchmark evaluation, and VLA post-training with avg@8 reporting.The study uses Qwen3.5-4B and Qwen3.5-9B, with results averaged over eight attempts.
  • 5.2. Main Results on AutoResearch Tasks: WMRL cuts real-execution GRPO training compute by 3.1× and 3.4× while scoring higher on every benchmark, with gains up to 3.1 points.Table 1 evaluates held-out tasks and reports leaderboard percentile scores with GPU-hours as training compute.
  • 5.2. Main Results on AutoResearch Tasks: The post-trained 4B and 9B agents surpass the 48B and 120B off-the-shelf agents, respectively, on both reported averages.The comparison tests whether model scale alone can replace post-training.
  • 5.3. Transfer to VLA Post-Training: WMRL transfers to VLA post-training, where a predicted progress reward is calibrated against sparse environment success signals.The VLA setup uses Robometer predictions over eight sampled frames and terminal success as the anchor.
  • 5.4. Ablation Study: Activating both corrections improves every ablation column by 2.9 to 4.8 points, exceeding either correction alone.Inverse-Variance Denoising alone adds 0.9 to 1.7 points, while Online Debiasing alone adds 2.2 to 2.8 points.

6. Conclusion

The work scales AutoResearch RL by replacing expensive environment execution with a world model and correcting its bias and noise using a small stream of real execution. These corrections reduce compute by three to four times while matching or exceeding full real-execution RL at two scales, with transfer to VLA post-training.

  • 3–4x lower training compute while matching or exceeding full real-execution RL at two scales is the paper’s central outcome.The conclusion also reports transfer to embodied VLA post-training.
  • World-model rewards replace expensive environment execution, while Online Debiasing and Inverse-Variance Denoising correct bias and noise through a small anchored real-execution stream.
  • The two corrections turn world-model training’s permanent error floor into a contracting term and reduce variance below either reward stream alone.
  • Transfer to VLA post-training suggests WMRL can scale reinforcement learning where execution, rather than generation, is the bottleneck.

B. Proofs

The appendix proves the claims of Section 4 in sequence, covering standard real-execution RL, world-model rewards, WMRL, and the supporting lemmas.

  • The appendix proves Section 4’s claims in story order, ending with a term-by-term comparison for WMRL.
  • Appendix B.1 derives the standard convergence term ε(T), B.2 proves Theorem 3, and B.3 proves Theorem 4.
  • Auxiliary lemmas are cited in the main proof sequence and collected with full statements and proofs in Appendix B.4.

B.1. Warm-up: convergence of standard RL

This warm-up establishes the convergence bound for GRPO trained with real execution rewards, using a clean estimator, one-step progress, gradient domination, and unrolling.

  • GRPO trained on real execution rewards satisfies the standard convergence guarantee under Assumption 2 and γ ≤ 1/(8L).
  • The real-execution estimator is unbiased up to κ = 1 − 1 and has conditional variance bounded by V_E.
  • L-smoothness supplies one-step progress, while gradient domination relates gradient magnitude to the optimality gap.
  • Unrolling the recursion over T iterations yields geometric decay plus a variance-dependent convergence term.

B.2. Proof of Theorem 3

Theorem 3 analyzes GRPO trained on world-model rewards by decomposing predicted-gradient estimates into clean, bias, and noise components, which add perturbation and variance terms to convergence.

  • Predicted-score gradients decompose as the clean estimate plus a bounded deterministic bias shift and a zero-mean noise fluctuation.
  • GRPO trained on world-model rewards satisfies a convergence bound containing geometric decay, a bias term, and increased variance under Assumption 2.
  • World-model noise increases variance to V_WM = V_E + 4M^2σ^2/n because the clean and noise components are uncorrelated.
  • The bias shift is bounded by 2MB and enters the convergence recursion as a constant perturbation.

B.3. Proof of Theorem 4 and the comparison

Theorem 4 shows that WMRL’s recalibration makes the bias contribution contract over time, while inverse-variance fusion reduces the variance contribution relative to standard world-model training.

  • The recalibration error decreases with accumulated anchor pairs, and T0 measures when residual recalibrated bias falls below the raw world-model bias.
  • Theorem 4 bounds WMRL’s perturbation by a constant floor and a geometrically decaying remainder, with the contracting bias term scaling as O(M2c_f^2/T).
  • For sufficiently large T, WMRL’s bias term is strictly below the Theorem 3 floor and converges to zero as T →∞.
  • Inverse-variance fusion reduces the variance term by the factor 1 + V_WM/V_E > 1, using the post-recalibration world-model variance.
  • Plug-in variance weights are second-order insensitive to ratio error, with observed excess below three percent at the calibration point.

B.4. Auxiliary lemmas

The auxiliary lemmas connect reward errors to gradient perturbations, establish convergence under biased and noisy estimates, and justify recalibration and inverse-variance fusion.

  • The perturbed-ascent lemma is the common engine behind Proposition 5, Theorem 3, and Theorem 4, handling extra variance and bounded or shrinking perturbations.
  • The convergence proof combines smoothness, gradient domination, and a telescoping recursion after separately bounding useful progress and estimator second moments.
  • Reward-level bias and noise decompose the world-model gradient into clean, bias, and noise components, yielding a bounded bias term and V_WM = (1 + cσ^2)V_E.
  • The recalibration lemma shows that isotonic fitting learns the score distortion at a 1/t rate under finite score levels, non-collapsed distortions, and adequate anchor coverage.
  • Inverse-variance weighting minimizes the variance among linear unbiased combinations and achieves the harmonic variance, strictly below either component variance.

C. Choice of the Task Pool

The task pool is built from MLE-Dojo to avoid unavailable competitions, problematic medal scoring, and contamination concerns, while providing a deterministic, category-balanced train/test split. The resulting pool contains 45 training and 14 held-out competitions, with additional exclusions for sandbox or grader constraints.

  • Motivation: The original pool has unusable competitions, medal-based scores that can remain flat as agents improve, and contamination risks from old competitions and winning solutions.Two competitions are permanently closed, medal counts cover only a small fixed subset at the studied scales, and the public evaluation has not been actively maintained.
  • Choice of MLE-Dojo: MLE-Dojo is preferred because it provides live grading, exposes leaderboard percentiles, and supports repartitioning, unlike the directly used MLE-Bench pool.The shared split is fixed once and used across configurations.
  • Why these training and test sets: 45 training and 14 held-out competitions are selected from MLE-Dojo using a deterministic, category-balanced split based on the smallest competitions in each category.Within each category, 20 competitions are selected; 15 go to training and 5 to held-out evaluation, with one held-out task later removed.
  • Exclusions: Audio competitions and one held-out task are excluded because they do not fit the sandbox or depend on an unavailable grading package.The remaining evaluation set contains 14 held-out competitions.
  • Pool composition: The final MLE-Dojo pool contains 19 binary-classification, 10 multi-class-classification, 23 regression, and 8 other-metric tasks, with no overlap against 45 training competitions checked by name matching.The category counts sum to 60 tasks in the broader pool, while the train/test split uses 59 competitions after one held-out task is dropped.

D. Experimental Details

Experiments train language agents and VLA policies with GRPO while comparing real execution against world-model-based training under controlled sandbox and prompting interfaces. The world model is prompted rather than fine-tuned, and calibration, debiasing, denoising, and anchor weighting address discrepancies between predicted and real outcomes.

  • AutoResearch training: GRPO training uses eight groups per step and eight trajectories per group, with up to four interaction turns, 4096 generated tokens per turn, and 1024-token observations.Optimization uses a 1 × 10^-6 learning rate, KL coefficient 0.04, entropy coefficient 0.002, temperature 1.0, and top-p 1.0.
  • Execution environment: AutoResearch agents operate in isolated Python 3.11 sandboxes with one approximately 40 GB GPU, a 1200-second budget, and a fixed environment contract for inputs and submissions.The sandbox provides the libraries listed in Table 7, and the agent must write its submission to SUBMISSION_PATH.
  • World-model interface: The world model receives the task description and current solution, simulates execution, and returns a score-related outcome without fine-tuning.Its context is capped at 12k tokens and its prediction at 1024 tokens.
  • Calibration and stability: Calibration fits a monotone correction after 200 anchor pairs and refits it every 64 new pairs, while the calibrated conversion factor is reported as 0.96.The anchor factor is bounded within [1, 4] for training stability, and the fused variance is described as second-order insensitive to the weight.
  • VLA setup: VLA experiments use MiniVLA-1B, supervised fine-tuning on 50 demonstrations per task, and 40 GRPO steps with 64 rollouts per task and a 520-step horizon.Training uses 16 official initial states, while evaluation uses all 50.
Loading 2608.12564v3…