PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Execution-Grounded Automated AI Research

Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candès, Diyi Yang, Tatsunori Hashimoto

arXiv:2601.14525v1cs.CLcs.AIcs.LG

TL;DR

LLMs often generate plausible but ineffective research ideas, leaving automated execution and learning from execution feedback unresolved. The paper builds an automated executor for realistic pre-training and post-training environments, then evaluates evolutionary search and reinforcement learning. Evolutionary search finds strong recipes efficiently, whereas reinforcement learning improves average reward but not the upper bound because ideas collapse toward simple implementations.

  • Problem

    Current LLMs often generate plausible-looking but ineffective research ideas, and it remains unclear whether automated execution and learning from execution feedback are feasible.

  • Method

    The paper builds a large-scale automated executor for LLM pre-training and post-training, then uses execution feedback with evolutionary search and reinforcement learning.

  • Results

    Execution-guided evolutionary search finds 69.4% versus 48.0% on post-training and 19.7 minutes versus 35.9 minutes on pre-training, while reinforcement learning improves average reward but not the upper bound.

  • Takeaways & Limitations

    Automated execution feedback is feasible for realistic open-ended LLM research, but effective learning still requires addressing limited scaling and diversity collapse.

  • Takeaways & Limitations

    In the current setup, reinforcement learning improves average reward but not the upper bound, likely because exploration and idea diversity remain insufficient.

Abstract

from arXiv · show

Automated AI research holds great potential to accelerate scientific discovery. However, current LLMs often generate plausible-looking but ineffective ideas. Execution grounding may help, but it is unclear whether automated execution is feasible and whether LLMs can learn from the execution feedback. To investigate these, we first build an automated executor to implement ideas and launch large-scale parallel GPU experiments to verify their effectiveness. We then convert two realistic research problems - LLM pre-training and post-training - into execution environments and demonstrate that our automated executor can implement a large fraction of the ideas sampled from frontier LLMs. We analyze two methods to learn from the execution feedback: evolutionary search and reinforcement learning. Execution-guided evolutionary search is sample-efficient: it finds a method that significantly outperforms the GRPO baseline (69.4% vs 48.0%) on post-training, and finds a pre-training recipe that outperforms the nanoGPT baseline (19.7 minutes vs 35.9 minutes) on pre-training, all within just ten search epochs. Frontier LLMs often generate meaningful algorithmic ideas during search, but they tend to saturate early and only occasionally exhibit scaling trends. Reinforcement learning from execution reward, on the other hand, suffers from mode collapse. It successfully improves the average reward of the ideator model but not the upper-bound, due to models converging on simple ideas. We thoroughly analyze the executed ideas and training dynamics to facilitate future efforts towards execution-grounded automated AI research.

1. Introduction

The paper builds an automated execution-feedback loop for open-ended LLM research and evaluates whether LLMs can generate and improve ideas through execution-grounded search and reinforcement learning. Evolutionary search finds strong recipes efficiently, while scaling is limited and reinforcement learning improves average reward without improving the upper bound.

  • Motivation: Automated AI research aims to have LLMs generate ideas, implement them as code, run experiments, and learn from execution results.This could convert compute into scientific discovery, but LLM-generated ideas often appear convincing yet prove ineffective after execution.
  • Approach: A high-throughput automated executor implements hundreds of natural-language ideas in parallel to provide scalable experiment feedback.The executor addresses the challenge of obtaining execution results for open-ended research ideas.
  • Feasibility: Over 90% execution rates on the pre-training environment show that Claude-4.5-Sonnet and Claude-4.5-Opus can implement a large fraction of generated ideas.The experiments use LLM pre-training and post-training as realistic GPU-intensive research environments.
  • Execution-guided search: Within ten search epochs, evolutionary search reaches 69.4% versus 48.0% for the GRPO baseline and 19.7 minutes versus 35.9 minutes for the nanoGPT baseline.The search also produces algorithmic ideas beyond hyperparameter tuning and significantly outperforms best-of-N under the same sampling budget.
  • Execution-guided search: Only Claude-4.5-Opus shows a clear scaling curve, while Claude-4.5-Sonnet and GPT-5 saturate early.This limits how consistently stronger models or larger search budgets translate into better ideas.
  • Reinforcement learning: Reinforcement learning improves average reward but not maximum reward, as the ideator converges on easy-to-implement ideas and loses thinking length and diversity.The upper bound is the more important metric for scientific discovery in this setting.

2. Automated Idea Executor

The automated idea executor converts natural-language research ideas into code changes, schedules parallel GPU experiments, and returns benchmark performance. It is grounded in fixed research environments for nanoGPT pre-training and GRPO post-training, with safeguards against reward hacking.

  • Executor overview: The executor takes natural-language research ideas as input, generates code implementations, runs backend experiments, and returns benchmark performance.Its environments provide a research problem, baseline codebase, benchmark, fixed data, and evaluation metrics.
  • Pre-training task: The nanoGPT environment minimizes time to train a 124M GPT-2 model on FineWeb until reaching 3.28 validation loss on 8 H100 GPUs.The task is based on a modified nanoGPT speedrun setting.
  • Post-training task: The GRPO environment measures maximum MATH validation accuracy for a Qwen2.5-Math-1.5B model under a fixed training wall-clock budget.The baseline is an implementation of GRPO, and validation-related code is kept inaccessible to the executor.
  • Research scope: The ideation scope includes both hyperparameter tuning and novel model architectures or training algorithms.No constraints are imposed on the kinds of ideas proposed in either environment.
  • System design: The system comprises an Implementer that applies code diffs, a Scheduler that allocates resources, and Workers that run GPU experiments and upload results.Successful runs include performance metrics and metadata, while failed executions halt.
  • Evaluation: The executor compares model-generated ideas against baseline performance in the GRPO and nanoGPT environments.The figure reports baseline GRPO accuracy of 0.480 and baseline nanoGPT loss of 3.255.

3. Benchmarking LLM Ideators and Executors

Benchmarking shows that frontier LLMs can serve as both ideators and executors, producing meaningful execution feedback in realistic research environments. Using different ideator and executor models remains feasible, although execution rates vary and some open-weight models outperform the baselines.

  • Benchmark setup: The benchmark evaluates frontier LLMs as ideators and executors using completion rate, average performance, and best performance across sampled ideas.The self-execution setting samples and executes 50 ideas from each model.
  • Cross-model execution: With GPT-5 as executor, execution rates range from 42% to 78% across ideator models.Claude-4.5-Sonnet’s execution rate falls from 84% to 42% on GRPO and from 90% to 78% on nanoGPT when GPT-5 executes its ideas.
  • Cross-model execution: Qwen3-235B reaches 50.2% maximum accuracy on GRPO and 3.238 minimum loss on nanoGPT with N = 50, outperforming both baselines.This demonstrates non-trivial best-of-N performance even when the executor is fixed to GPT-5.
  • Implication: The benchmarking results support the feasibility of an automated ideation-and-execution loop and motivate search scaffolds and reinforcement-learning training loops.The next stage tests whether models can learn from the resulting execution feedback.

4. Execution-Guided Evolutionary Search

Execution-guided evolutionary search combines exploration of new ideas with exploitation of successful prior trajectories, using execution feedback to optimize open-ended research ideas. Across nanoGPT and GRPO, it improves over best-of-N and baselines, while model scaling and idea diversity vary substantially.

  • Search Method: Execution-guided search uses evolutionary optimization without gradient updates, combining exploration with exploitation of ideas that outperform the baseline.Subsequent epochs prompt the ideator with successful prior ideas while retaining an exploration subset of newly sampled ideas.
  • Model Scaling: Claude-4.5-Opus shows a scaling trend as search epochs increase, whereas Claude-4.5-Sonnet and GPT-5 tend to saturate early.The comparison evaluates Claude-4.5-Opus, Claude-4.5-Sonnet, and GPT-5 as both ideator and executor.
  • Performance: 19.7 minutes versus 35.9 minutes: Claude-4.5-Opus’s nanoGPT recipe reaches the target validation loss faster than the baseline.The recipe combines architectural modifications, hyper-parameter tuning, and exponential moving averages of intermediate checkpoints during validation.
  • Comparison with Best-of-N: Evolutionary search significantly outperforms best-of-N from epoch 1 onward under the same sampling budget of N=80 ideas per epoch.Both methods begin with similar performance at epoch 0, while search uses trajectories from previous epochs to generate later ideas.
  • Analysis of Generated Ideas: All three models generate substantial algorithmic ideas beyond hyper-parameter changes, and algorithmic ideas are most effective in most cases except for Claude-4.5-Sonnet.The analysis classifies ideas as hyper-parameter tuning or algorithmic changes across GRPO and nanoGPT searches.

5. Reinforcement Learning from Execution Reward

The paper evaluates reinforcement learning from execution reward for open-ended AI research, finding that it raises average reward but not the upper-bound because training converges on simple, low-diversity ideas.

  • Setup: RL from execution reward fine-tunes Qwen3-30B with GRPO, using executed idea performance as reward in the GRPO and nanoGPT environments.The experiments use group sizes of 256 for post-training and 128 for pre-training, with extracted ideas executed in parallel.
  • Average reward: 0.343 average accuracy after 40 epochs versus 0.253 initially in GRPO, while nanoGPT reward rises from 0.194 to 0.246 after 68 epochs.The nanoGPT increase corresponds to average validation loss decreasing from 5.150 to 4.066.
  • Max reward: The max reward fluctuates without a clear upward trend, unlike average reward, making upper-bound improvement the central limitation for scientific idea generation.The paper prioritizes finding breakthrough ideas over producing many safe ideas with high average quality.
  • Training dynamics: Longer thinking traces have lower execution rates, and thinking length decreases during RL while idea length stays roughly constant.The authors hypothesize that longer thinking correlates with more complex, harder-to-execute ideas, encouraging shorter thinking to maximize reward.
  • Training dynamics: RL causes diversity collapse as the model converges on a few easy ideas, including replacing RMSNorm with LayerNorm and applying EMA to intermediate checkpoints.The model’s convergence on these two ideas is shown in the nanoGPT environment.
  • Interpretation: Convergence on simple ideas raises average reward but fails to improve the upper-bound because exploration is lost.The paper identifies avoiding this collapse as an open problem beyond standard GRPO.

6. Related Work

The work relates to AutoML, LLM-based research agents, AI-for-research components, and execution-grounded code generation while differing through open-ended idea generation and expensive research verification.

  • AutoML: Unlike constrained AutoML, this work searches over unrestricted natural-language research ideas rather than predefined architectures or code optimizations.Its goal is to improve idea generation at a higher abstraction level than specific architecture variants.
  • LLM-based Research Agents: LLM-based research agents also target open-ended AI research, but prior systems use carefully designed end-to-end agent scaffolds for idea generation and implementation.The cited examples include AI-Scientist, AI-Researcher, and Agent Laboratory.
  • AI for Research: AI-for-research studies address individual pipeline components such as literature review, idea generation, data analysis, experiment planning, and code execution.These component-focused efforts differ from the paper’s integrated execution-grounded study of idea generation.
  • Execution Grounding for Code: Execution feedback has guided code refinement and code-generation training, whereas this work applies execution grounding to idea generation with more complicated and expensive verification.The distinction is the research-level action space and evaluation cost.

7. Conclusion

The paper builds a parallel executor for open-ended LLM research and uses it to compare evolutionary search with reinforcement learning from execution rewards. Evolutionary search outperforms baselines, whereas reinforcement learning improves average reward but sacrifices diversity without improving the upper-bound.

  • Conclusion: The system automatically executes model-generated ideas for open-ended LLM pre-training and post-training problems.The executor provides the experimental feedback used throughout the study.
  • Conclusion: Execution-guided evolutionary search enables frontier LLMs to significantly outperform baseline solutions with a simple search scaffold.The conclusion presents this as evidence for the feasibility and potential of automated execution feedback.
  • Conclusion: Reinforcement learning with execution rewards converges on simple ideas, loses diversity, and improves average reward without improving the upper-bound.The conclusion identifies these as remaining limitations for future work.

8. Discussion

The experiments have limited scope: they do not test idea generalization, RL improves average reward but not the upper bound, execution-agent capability constrains coverage, and effectiveness is the only reward explored.

  • Generalizability: The procedure does not test whether ideas that perform well at small scales transfer to larger scales or other datasets.Future work is proposed to test and optimize for generalizability and scalability.
  • RL limitations: RL with execution reward improves average reward but not the upper bound in the current setup.The passages identify limited base-model diversity and missing exploration incentives as possible reasons.
  • Execution constraints: The experiment scope is bounded by the execution agent, and unexecuted ideas introduce noise into the reward signal.More capable coding agents with external tools and library-installation access are proposed as remedies.
  • Reward scope: The study uses effectiveness as its training reward, leaving novelty and interestingness for future computational measurement.The paper suggests incorporating such metrics into the training objective.

A.1. Other RL Attempts

The appendix evaluates several attempts to improve reinforcement learning from execution reward, including dynamic prompts, length rewards, and diversity rewards.

  • Dynamic Prompt: Dynamic prompting appends randomly sampled trajectories from the previous epoch to new idea-sampling prompts.The approach combines in-context learning with RL and adds diversity to sampling, but was early-stopped after no significant early improvement.
  • Length Reward: A capped length reward counts tokens in the thinking trace and idea to counter the rapid decrease in thinking length.The cap is 0.3 to avoid the length reward dominating accuracy reward.
  • Reported evaluations: The appendix reports Figure 8 for dynamic prompts and Figure 9 for length rewards on the GRPO environment.Figure 10 reports effectiveness reward and average idea similarity when using a diversity reward.
  • Diversity Reward: A diversity reward penalizes token-level Jaccard similarity between current ideas and ideas from the previous epoch.The penalty is intended to discourage repeating previously generated ideas.

A.2. Additional Idea Examples

The additional examples show both execution failures and successfully executed pre-training ideas, ranging from extensive hyperparameter recipes to architectural modifications and optimizer changes.

  • Execution failures: Some model-generated ideas fail because they require complicated changes or unsupported external packages.The paper identifies more capable execution agents as a direction for implementing auxiliary models and system-level optimizations.
  • GRPO examples: The appendix presents additional GRPO-environment examples generated by Claude-4.5-Opus and Claude-4.5-Sonnet.These examples include ideas with failed code execution.
  • Claude-4.5-Opus recipe: A Claude-4.5-Opus nanoGPT recipe combines wider SwiGLU, learned skip connections, residual scaling, optimizer changes, EMA, and schedule modifications.The recipe is reported with validation loss 3.1407.
  • Architecture changes: The examples also include architectural changes such as wider MLPs, skip connections, separate residual scales, untied embeddings, and EMA.The listed code changes implement these modifications across GPT and MLP components.
  • Training recipes: A second nanoGPT recipe combines two-phase weight decay, a plateau, warmup, gradient clipping, cosine learning-rate decay, and progressive EMA.Its specified settings include weight decay 0.1170→0.0210 and EMA 0.999→0.9992.
  • Optimization changes: Additional recipes modify learning-rate schedules, beta2, gradient clipping, weight decay, and EMA updates during training and evaluation.The appendix provides the corresponding implementation changes for these schedules and state-management procedures.
  • Additional successful ideas: The appendix reports a GPT-5 nanoGPT idea with validation loss 3.1697 and describes other successfully executed atomic algorithmic ideas.These examples are contrasted with heavily optimized recipes combining hyperparameter tuning and architecture changes.

Examples from Claude-4.5-Opus

Claude-4.5-Opus generated diverse nanoGPT ideas spanning attention, residual pathways, embeddings, normalization, and token-level objectives, with several reported validation losses.

  • Attention and depth: Head-wise attention output scaling adds learnable per-head factors, with validation loss 3.2386.The scaling allows different attention heads to contribute with different magnitudes.
  • Reported validation losses: Learned residual connection weights achieve validation loss 3.2517, while mixture of embeddings with position achieves 3.2497.Both ideas introduce learned mechanisms to adapt representation processing during training.
  • Reported validation losses: Shared input-output embeddings with learned asymmetry, gated final normalization, and position-aware MLP gating achieve validation losses 3.2499, 3.2503, and 3.2506.These ideas modify output adaptation, normalization strength, and position-dependent MLP processing.
  • Attention and depth: Soft layer repetition uses a learned gate to mix a layer’s input back into its output, simulating variable depth.The passage describes this as softly repeating computation through layers.
  • Attention and depth: Causal context compression mixes the previous 2–3 tokens into the current representation before each attention layer.The proposed mechanism provides implicit local context without convolutions.
  • Attention and depth: Attention-head specialization adds an orthogonal loss that penalizes similarity between head outputs.The objective encourages different heads to attend to different patterns.
  • Residual pathways: Skip connections with learned residual weights provide alternative gradient paths while allowing adaptive scaling.The idea combines skip connections with learned residual weights.
  • Training objectives: Token difficulty-aware loss weighting emphasizes uncertain tokens while avoiding over-optimization of easy tokens.The weighting uses model entropy at each token position.

A.3. Code Execution Examples

The examples show frontier-model ideas being translated into executable code for GRPO-based mathematical problem solving. Implementations add mathematical context, extract expressions and definitions, retrieve relevant items, and feed enhanced prompts into rollout and training.

  • Execution examples: Claude-4.5-Sonnet examples pair each natural-language idea with generated code executed in the GRPO environment.The examples are presented as full end-to-end trajectories from idea through implementation.
  • Mathematical context: A mathematical working-memory idea maintains facts, definitions, and intermediate results in a context buffer for later problem-solving steps.The buffer is updated during problem solving and supplies additional context for subsequent steps.
  • Mathematical context: The implementation extracts mathematical expressions and definitions from responses, removes duplicates, and inserts selected context items into enhanced prompts.Extraction covers inline math, LaTeX commands, fractions, square roots, and several definition-like patterns.
  • Mathematical context: A MathContextBuffer retains recent items, scores them by shared mathematical terms with the query, and returns the most relevant context.The implementation considers recent buffer entries and returns the highest-scoring items, up to a specified maximum.
  • Training integration: The training loop samples rollout batches with context-enhanced prompts, computes rewards, and updates the context buffer from generated responses.The modified loop creates enhanced prompts for each batch, samples responses and rewards, and extracts mathematical expressions for buffer updates.
  • Additional experiment: A separate robustness idea trains on mathematically equivalent problems with varied presentations and uses robustness-aware rewards to encourage generalization across variations.The proposal combines systematic perturbation testing with reward shaping.
Loading 2601.14525v1…