PaperlayerView exact revision on arXiv

Source-linked AI summary

Verify Less, Evolve More: Training Idea-Level Critics for Verification-Efficient ML Evolving Agents

Jiamu Bai, Lizhu Zhang, Xin Yu, Yanhong Wu, Zellux Wang, Serena Li, Weiwei Li, Zhuokai Zhao, Lingzhou Xue, Kiwan Maeng, Xiangjun Fan, Bo Peng

arXiv:2610.08993v1cs.AI

TL;DR

Costly empirical verification limits the scale of ML-agent evolution, and frontier models used directly as idea selectors provide only limited gains. The paper trains idea-level critics to screen proposals, then reports improvements in static evaluation and agent evolution, including 2.75× as many policy-optimization steps under the same verification budget.

  • Problem

    In AI4ML, empirical verification through model training and evaluation is computationally costly, while directly prompting a frontier model as an idea selector yields only modest gains.

    Source: §1 · p. 1

  • Method

    The paper trains idea-level critics on soft-rubric critiques with supervised fine-tuning and GRPO, then uses their predictions for idea selection and as rewards for policy training.

    Source: §1 · pp. 1–2

  • Results

    2.75× as many policy-optimization steps are performed under the same verification budget, alongside reported gains in static evaluation, inference-time selection, and continual learning.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Idea-level critics provide a practical interface between idea generation and expensive execution, helping ML agents use limited verification budgets more effectively.

    Source: §7 · p. 10

Abstract

from arXiv · show

As large language models become more powerful, self-evolving agents are able to tackle challenging tasks including AI for machine learning (AI4ML). In AI4ML, while empirical verification is available, it often requires computationally costly model training and evaluation, limiting the speed and scale of agent evolution. Yet verification efficiency remains under-explored, and frontier models provide only limited gains when used directly as idea selectors. We address this gap with specialized idea-level critic models that predict whether a proposed ML modification will improve upon the current solution, allowing agents to screen ideas and concentrate verification resources on the most promising candidates. We train the critic models through supervised fine-tuning on high-quality critiques synthesized by Gemini-3.1-Pro, followed by GRPO to further improve their predictive accuracy. Empirically, our critic models outperform Gemini-3.1-Pro in static idea evaluation, and these gains extend to agent inference, continual learning, and policy training. During inference-time evolution, they improve final solution quality under the same verification budget by selecting more promising ideas, with further gains from continual learning. During policy training, they serve as learned reward models, reserving empirical verification for uncertain cases and enabling substantially more policy updates with the same verification resources. Together, these results show that idea-level critic models help ML agents discover better solutions and learn stronger proposal policies under limited verification budgets.

1 Introduction

Costly empirical verification limits how many ML modifications agents can explore, while prompting a frontier model as an idea selector offers only modest gains. The paper trains idea-level critics to screen proposals and reports gains in static evaluation, inference-time evolution, continual learning, and policy training.

1 Introduction

  • The training pipeline uses real ML agent evolution trajectories and Gemini-3.1-Pro-synthesized critiques filtered for consistency with execution-derived labels.The critics are trained with supervised fine-tuning followed by GRPO.Source: §1 · pp. 1–2
  • Idea-level critics predict whether a proposed modification will improve on the current solution, allowing agents to reserve costly empirical verification for promising candidates.Empirical model training and evaluation costs accumulate across candidates and evolutionary rounds, limiting exploration under a fixed compute budget.Source: §1 · p. 1
  • 60.7% and 63.3% three-way directional accuracy for the 9B and 35B critics, respectively, exceeds Gemini-3.1-Pro’s 46.6% on static idea evaluation.The critics are trained with soft-rubric critiques and further GRPO alignment.Source: §1 · p. 2
  • Critic-guided policy training enables 2.75× as many policy updates under the same verification budget by routing uncertain predictions to empirical verification.The critic serves as a learned reward model for GRPO training of the idea-proposing policy.Source: §1 · p. 2
  • In OpenEvolve, critic screening improves final solution quality under matched verification budgets, and continual learning further improves the final solutions.The critic selects promising proposals before empirical verification; directly prompting Gemini-3.1-Pro as an idea selector yields only limited gains.Source: §1 · p. 2

2 Building Criterion-Guided Idea-Level Critic Models

The section defines an ML evolution loop and adds a criterion-guided critic that predicts execution outcomes from task and solution descriptions. It describes the trajectory data and critique synthesis used to train the critic through supervised fine-tuning and GRPO.

2.1 Task Setting

  • The agent proposes, implements, and empirically evaluates child solutions, retaining a child only when it outperforms its parent; verification can require costly model training and evaluation.The critic is introduced to predict verification outcomes from idea-level descriptions before execution.Source: §2.1 · p. 3

2.2 Criterion-Guided Critic Model Design

  • Soft-rubric reasoning compares parent and child under every task-specific criterion, then allows flexible weighting and aggregation into a final verdict.Task-specific criteria are generated once per task from analysis of its setup, data, metric, and modeling constraints.Source: §2.2 · p. 4

2.3 Training Data and ML-Idea-Critic-Bench Construction

  • The dataset combines execution-labeled parent–child trajectories with critiques synthesized in three assistance strata: Sampled, Revised, and Guided.The SFT/evaluation pool uses five outcome labels, while the GRPO pool retains three-way labels and is balanced across classes.Source: §2.3 · p. 4

2.4 Critic Model Training

  • Training first uses approximately 11K Gemini-generated critiques for SFT, then applies GRPO to 15.5K additional execution-labeled examples with reward based on three-way verdict accuracy.The remaining 1.5K examples form the static evaluation set; SFT teaches criterion-level analysis and final verdicts.Source: §2.4 · pp. 4–5

3 ML-Idea-Critic Bench Result

The benchmark evaluates idea critics across four complementary metrics and compares trained critics with frontier, base-model, and reward-model baselines. GRPO improves critic accuracy, while soft-rubric reasoning remains stable under reinforcement learning where freestyle reasoning deteriorates.

3 ML-Idea-Critic Bench Result

  • The benchmark reports Improve Accuracy, three-way Dir-3 accuracy, F1 for binary improvement, and exact-match accuracy across the five-level 5-Ladder scale.The listed comparison set includes Gemini-3.1-Pro, base Qwen models, and DeepSeek-GRM models.Source: §3 · p. 5
  • GRPO raises Dir-3 accuracy to 60.7% for the 9B critic and 63.3% for the 35B critic, versus 46.6% for Gemini-3.1-Pro.These are gains of 5.1% and 3.0% over their SFT checkpoints, respectively.Source: §3 · p. 5
  • Under GRPO, soft-rubric reasoning reaches 60.7% Dir-3 accuracy, while freestyle reasoning falls to 40.3%.After SFT, the formats score 55.6% and 54.6%, respectively; the authors report that criterion-by-criterion comparisons stabilize reinforcement learning.Source: §3 · p. 5

4 Integration with Self-Evolving Agents

The authors integrate idea critics into OpenEvolve to screen proposals before empirical verification and adapt critics using feedback gathered during evolution. Trained critics improve solution quality under matched verification budgets, and SDFT improves downstream evolution more consistently than test-time GRPO.

4.1 Inference-Time Idea Selection

  • OpenEvolve tests seven screening settings on 29 MLS-Lite-Bench tasks across 11 domains, using three islands, 10 rounds per island, and 30 verification runs per task.For each of 16 proposed ideas, the critic produces 16 critiques; only the idea with the most win votes is verified.Source: §4.1 · pp. 5–6
  • Trained critics achieve the highest overall average and the best score in 9 of 11 domains; on 20 tasks excluded from critic training, Critic-9B and Critic-35B average 34.24 and 34.18, versus 30.44 without screening.The largest reported gain is in Language Models, where the normalized score rises from 18.04 without screening to 32.57 with Critic-9B; all settings share Gemini-3.1-Pro as proposer and implementer.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 Test Time Continual Learning

  • Test-time SDFT updates the critic after each OpenEvolve round from three newly verified parent–child pairs, while GRPO updates every two rounds from six pairs across three islands.SDFT uses parent and verified-child raw scores as privileged teacher information and minimizes token-level reverse-KL loss along the student critique trajectory; GRPO rewards each of 16 rollouts by whether its three-way prediction matches the empirical label.Source: §4.2 · pp. 6–7
  • SDFT improves downstream evolution more consistently than test-time GRPO under a fixed verification budget.The authors attribute a richer adaptation signal to SDFT's outcome-conditioned token-level targets, compared with GRPO's binary correctness rewards; both adaptations use already-performed verifications without extra verifier calls.Source: §4.2 · p. 7, Table 3 · p. 7

5 The Critic Serving as Reward Model

Critic predictions can serve as policy-training rewards, routing uncertain ideas to empirical verification and allowing more updates under a fixed verification budget. The critic-guided policy improves normalized domain scores over verify-all GRPO, while uncertainty analysis and ablations probe the routing trade-off and reward-model behavior.

5 The Critic Serving as Reward Model

  • Critic-guided selection remains constrained by the fixed policy model’s ability to propose ideas.The section motivates using critic predictions as training signals to improve proposal capability.Source: §5 · p. 7

5.2 Policy Model GRPO Result

  • 2.75× more candidate ideas received training rewards per ground-truth verification with critic-guided assignment than verify-all under the same 10,000-verification budget.Only 36.3% of policy-generated ideas required empirical verification, enabling about 110 steps rather than 40; this ratio does not measure wall-clock speedup or total compute savings.Source: §5.2 · p. 8
  • Average normalized domain scores rose from 19.57 to 25.21 for Qwen-3.5-9B and from 19.29 to 24.55 for Qwen-3.5-35B with critic-in-loop versus verify-all GRPO.The authors suggest saved verification budget supports additional policy updates and shifts the policy toward more promising modifications.Source: §5.2 · p. 8

5.3 Ablation Experiments

  • At k = 16 and T = 0.25, uncertainty routing recalled about 70% of critic errors while sending about 30% of ideas to the verifier; AUROC gains largely saturated at k = 16.The critic-in-loop experiments verified 36.3% of policy-generated ideas, moderately above the benchmark estimate.Source: §5.3 · p. 9, Figure 4 · p. 9
  • With frozen Critic-35B rewards and no real verification, policy performance generally improved with more training steps on two selected tasks, while gains from increasing k were modest and saturated beyond 16.Increasing k beyond 16 proportionally increases inference computational cost.Source: §5.3 · p. 9

6 Related Work

Prior work studies self-evolving agents, ML research automation, idea evaluators, and generative critics. This work focuses on execution-grounded evaluation of proposed modifications to existing ML solutions, using the critic for candidate selection and policy training.

Self-Evolving Agents and ML Agents

  • Self-evolving ML agents use empirical feedback, while existing approaches improve fixed-model search harnesses or learn proposal policies; this work complements them by evaluating modifications before costly execution.The related-work discussion also frames ML agents as iterative propose–implement–evaluate systems.Source: §6 · p. 9

AI for Machine Learning Research

  • Unlike pretrained evaluation modules that prioritize candidates, this work makes the evaluator the learned component, predicting the empirical value of proposed ML modifications.Related work also covers AI research systems, paper reviewing, and research-idea evaluation.Source: §6 · p. 9

Critic Models and Generative Reward Models

  • Generative critics provide textual assessments for output evaluation, refinement, and ranking, while rubric-guided methods structure judgments across multiple criteria.Conventional reward models typically assign scalar scores; this work evaluates proposed modifications to existing ML solutions whose value requires empirical execution.Source: §6 · pp. 9–10

7 Conclusion

The conclusion presents idea-level critics as a feedback mechanism for self-improving ML agents, with applications in candidate selection, continual learning, and policy optimization. The related-work and data-collection passages situate this approach and describe how its task environments and labeled examples are constructed.

7 Conclusion

  • Idea-level critics combine soft-rubric reasoning, SFT, and GRPO, and support inference-time selection, continual learning, and critic-guided policy optimization.The conclusion reports 2.75× more policy-optimization steps under the same budget and presents critics as an interface between idea generation and expensive execution.Source: §7 · p. 10

A More Related Work

  • Related work spans self-evolving ML agents, AI research automation, paper-review systems, and generative critics; this work trains an execution-grounded critic on empirical parent–child outcomes for selection and policy optimization.The related-work discussion distinguishes evaluation of proposed modifications to existing solutions from research-idea evaluation and from critics focused on language or reasoning outputs.Source: A More Related Work · pp. 15–16

B.1 Task Environments and Initial Parent Solutions

  • For MLS-Bench tasks, existing source-code baselines become parent solutions; for selected MLE-Dojo tasks, Gemini-3.1-Pro generates and executes runnable initial solutions.The initial MLE-Dojo parent score comes from execution in its task environment.Source: §B.1 · p. 16

B.2 Trajectory Collection and Label Construction

  • The SFT/evaluation pool uses children proposed and executed from retained parents, while a separate iterative GRPO pool follows improving children as new parents and balances outcome classes by downsampling.For the SFT/evaluation pool, each retained parent receives 20 independently proposed and executed children; the GRPO pool begins with about 25K examples and is downsampled to 15,669 balanced examples.Source: §B.2 · p. 16

B.3 Parent-Specific Label Thresholds, Dataset Statistics, and Label Distribution

Parent-specific labels combine absolute and standardized improvement thresholds to account for differences in parent quality and difficulty. The section also describes the critic-training setup and an example of how a candidate moves through the ML-agent pipeline.

B.3 Parent-Specific Label Thresholds, Dataset Statistics, and Label Distribution

  • Absolute relative-improvement thresholds capture large gains, while standardized thresholds adapt labels to each parent’s local improvement difficulty.The standardized score uses a Gaussian approximation; the final five-level labels are assigned deterministically.Source: §B.3 · pp. 16–17
  • The SFT labels comprise 25.4% BigWin, 17.5% MediumWin, 15.5% SmallWin, 13.6% Tie, and 28.0% Lose.Source: §B.3 · p. 17

C.1 ML Agent Pipeline Examples

  • The running example follows a candidate from proposal and implementation through critic screening and empirical verification.The implementation replaces only the editable CustomActivation class, and its code-grounded summary becomes the critic’s child input.Source: §C.1 · p. 17
  • The activation-function task keeps architectures, optimizer, and data pipeline fixed; its ReLU parent has a combined_score of 0.5997.Source: §C.1 · p. 17

C.2 Task understanding.

  • The critic receives precomputed task context covering the evaluation objective, interface, datasets, input shapes, and preprocessing, reused across candidates.Source: §C.2 · p. 21

E Soft-Rubric Reasoning Examples

  • The example contrasts freestyle reasoning focused on overfitting risk with soft-rubric reasoning that compares the parent and child under task-specific criteria.Source: E Soft-Rubric Reasoning Examples · p. 22

F.1 Supervised Fine-Tuning Configuration

  • Both critic model sizes are trained for three epochs using LoRA with rank 32 and a learning rate of 1 × 10−5.Source: §F.1 · p. 22

D Complete Prompt Instructions

  • The critic prompt requires consideration of task-specific criteria while allowing the model to choose their relative importance and aggregation.Source: Table 9 · p. 23

F.3 Policy GRPO Training Details

The section describes policy GRPO training with either verification of every candidate or critic-guided reward assignment. The supplied ablation results report additional gains from criterion-guided GRPO, while the qualitative comparison distinguishes soft-rubric reasoning from freestyle reasoning.

F.3 Policy GRPO Training Details

  • Policy GRPO uses 16 parent groups with 16 candidate ideas each and compares verify-all training with critic-in-loop reward assignment.The policy is initialized from Qwen-3.5-9B and Qwen-3.5-35B and trained across 95 MLE-Dojo and 46 MLS-Bench environments.Source: §F.3 · p. 23
  • Critic-in-loop checkpoints train for around 110 steps versus around 40 for verify-all checkpoints, targeting the same 10,000-verification budget.Source: §F.3 · p. 23

G.1 Full Freestyle and Soft-Rubric Ablation Results

  • The qualitative example contrasts freestyle reasoning centered on overfitting risk with soft-rubric reasoning that compares task-specific criteria before aggregating judgments.Source: Table 10 · p. 24
  • Criterion-guided GRPO provides consistent additional gains over SFT for both model sizes on the 1.5K-example evaluation set.Source: Table 11 · p. 24

G.2 Results across Sampled, Revised, and Guided Examples

The section reports critic performance across three subsets with increasing teacher assistance and describes the evolutionary-search setting used for candidate screening. In that setting, one selected idea per round receives empirical verification.

G.2 Results across Sampled, Revised, and Guided Examples

  • Table 12 reports critic performance across Sampled, Revised, and Guided subsets, which represent increasing levels of teacher assistance during critique collection.Source: §G.2 · p. 24, Table 12 · p. 25

H.1 OpenEvolve Framework

  • OpenEvolve maintains a population of evaluated solutions and uses empirical feedback to accumulate improvements over successive rounds.Its MAP-Elites archive and island populations preserve diverse search trajectories and allow high-performing solutions to migrate between islands.Source: §H.1 · p. 25
  • Each task and screening setting uses three islands for 10 rounds, with one selected idea verified per island round for 30 verification runs total.The settings share the seed, proposer, implementer, summarizer, population configuration, and verification budget, differing only in candidate screening.Source: §H.1 · p. 25

H.2 Critic-Guided Candidate Selection

  • Each round ranks 16 proposed ideas using 16 critiques per idea and sends only the highest-average-ranked candidate for execution.Thus, one candidate consumes a real verification run while the other 15 are filtered without executing their training pipelines.Source: §H.2 · p. 25

H.3 Task Descriptions

The evaluation uses the official 30-task MLS-Bench-Lite subset and its task-specific normalized scoring specifications. It retains tasks that overlap with critic training environments; a separate qualitative comparison cautions that outcome-conditioned critiques do not validate causal explanations.

H.3 Task Descriptions

  • MLS-Bench-Lite is the official 30-task subset spanning all 12 MLS-Bench domains, with tasks covering varied ML development problems.Source: §H.3 · p. 26
  • Domain-level performance averages normalized task scores using the official task-specific scoring specifications without modification.Source: §H.3 · p. 26
  • Some MLS-Bench-Lite tasks also appear in critic training environments, and the authors retain them to preserve the complete official suite and its cross-domain composition.Source: §H.3 · p. 26

H.4 OPSD Teacher vs Student Critique Reasoning Trajectory

  • Table 14 shows outcome conditioning can shift a critique toward a mechanism consistent with an observed improvement, but the qualitative comparison does not independently validate that explanation.Source: §H.4 · p. 27

I.2 Ablation on the Critic Model as Reward Model

This ablation tests whether a frozen critic can supply policy rewards without empirical verification; performance generally rises with more training, while gains from additional critic rollouts are modest and begin to saturate.

I.2 Ablation on the Critic Model as Reward Model

  • The ablation uses frozen Critic-35B rewards without verifier fallback; Table 15 compares base, verify-all, and critic-in-the-loop GRPO under 10,000 empirical verifications.Table 15 reports averages across 11 retained MLS-Lite-Bench domains for Qwen-3.5-9B and Qwen-3.5-35B.Source: §I.2 · p. 27, Table 15 · p. 28
  • Beyond k = 16, additional rollouts provide limited benefit while increasing inference cost, making k = 16 the practical trade-off used.The stated trade-off balances reward reliability against critic inference computational cost.Source: §I.2 · p. 28
  • Policy performance generally improves with more training steps on ML Clustering and Graph Generation; increasing critic rollouts improves AI4Bio and Graph Generation, with gains beginning to saturate.Figure 5 varies training steps in the left panels and critic rollouts k in the right panels.Source: §I.2 · pp. 27–28, Figure 5 · p. 28
Loading 2610.08993v1…