PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

Alexis Audran-Reiss, Jordi Armengol-Estapé, Karen Hambardzumyan, Amar Budhiraja, Martin Josifoski, Edan Toledo, Rishi Hazra, Despoina Magka, Michael Shvartsman, Parth Pathak, Justine T Kao, Lucia Cipolina-Kun, Bhavul Gauri, Jean-Christophe Gagnon-Audet, Emanuel Tewolde, Jenny Zhang, Taco Cohen, Yossi Adi, Tatiana Shavrina, Yoram Bachrach

arXiv:2511.15593v2cs.AI

TL;DR

AI research agents are promising, but the factors behind their successes and failures remain poorly understood. The paper analyzes trajectories and intervenes on ideation diversity in MLE-bench; it reports that diversity relates to performance, reducing it lowers performance, and findings persist across evaluation metrics.

  • Problem

    The factors driving AI research agents’ successes and failures remain poorly understood, while complex trajectories and costly large-scale analyses make them difficult to study.

    Source: §1 · p. 2

  • Method

    The authors analyze 11,000 trajectories and measure architecture-choice entropy, then test diversity through a controlled prompt ablation on MLE-bench.

    Source: §1 · p. 2

  • Results

    The analysis and controlled experiment support ideation diversity as a performance bottleneck, with findings holding across several evaluation metrics.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    For agents with the same capability level, greater ideation diversity is preferred because it can de-risk implementation pitfalls.

    Source: §5 · p. 10

    Medal-rate comparisons are imperfect because thresholds vary by competition, custom and private test splits introduce score variability, and some contests are old.

    Source: §5 · p. 10

Abstract

from arXiv · show

AI research agents offer the promise to accelerate scientific progress by automating the design, implementation, and training of machine learning models. However, the field is still in its infancy, and the key factors driving the success or failure of agent trajectories are not fully understood. We examine the role that ideation diversity plays in agent performance. First, we analyse agent trajectories on MLE-bench, a well-known benchmark to evaluate AI research agents, across different models and agent scaffolds. Our analysis reveals that different models and agent scaffolds yield varying degrees of ideation diversity, and that higher-performing agents tend to have increased ideation diversity. Further, we run a controlled experiment where we modify the degree of ideation diversity, demonstrating that higher ideation diversity results in stronger performance. Finally, we strengthen our results by examining additional evaluation metrics beyond the standard medal-based scoring of MLE-bench, showing that our findings still hold across other agent performance metrics.

1 Introduction

The paper studies whether ideation diversity helps explain AI research-agent performance, addressing the difficulty of analyzing complex trajectories and measuring diversity. Across large-scale trajectory analysis and a controlled experiment, it reports that diversity is associated with performance and that reducing it lowers performance.

1 Introduction

  • The study examines research agents as an emerging approach to autonomous computational scientific discovery, spanning areas such as chemistry and biology.These systems aim to conduct independent research through a structured pipeline from idea generation to iterative refinement.Source: §1 · pp. 1–2

1.1 Contributions

  • The study analyzes 11,000 trajectories across multiple agent frameworks, LLM backbones, and machine-learning tasks.The trajectory bank comprises about 1.2 million search nodes and 264,000 GPU hours.Source: §1 · p. 2, §1.1 · p. 2
  • The authors propose quantifying ideation diversity with Shannon entropy over the distribution of model architectures agents plan to implement.Their trajectory-bank analysis shows a correlation between ideation diversity and MLE-bench performance.Source: §1 · p. 2, §1.1 · p. 2
  • A controlled prompt ablation that reduces ideation diversity also decreases agent performance on MLE-bench, and the findings hold across additional evaluation metrics.The controlled experiment modifies prompts on a subset of tasks; the paper also reports robustness beyond medal-based scoring.Source: §1 · p. 2, §1.1 · p. 2
  • Agent scaffolds significantly influence ideation diversity, which is significantly correlated with agent performance on MLE-bench tasks.The paper frames diversity as a potential bottleneck and identifies large-scale trajectory analysis and measuring and controlling diversity as key challenges.Source: §1.1 · p. 2, §1 · p. 2

2 Research Agents and MLE-bench

The section situates AI research agents as systems that use language models and an outer scaffold to conduct machine-learning work. It introduces MLE-bench and illustrates an iterative solution-search workflow.

2 Research Agents and MLE-bench

  • An AI research agent combines a model backbone that emits text actions with an outer loop, or scaffold, that interfaces with the environment.Agents can create and run code among other tools.Source: §2 · p. 3
  • Figure 2 follows an agent attempting a toxicity-classification task: it tries end-to-end fine-tuning, fixes a code failure, then continues improving the solution.The example depicts the agent producing additional search nodes as it proceeds.Source: Figure 2 · p. 3
  • MLE-bench comprises 75 Kaggle-derived machine-learning tasks across computer vision, NLP, forecasting, tabular, and multimodal domains.Tasks include documentation, training data, held-out tests, submission formats, and automated evaluation; agents work within computational constraints.Source: §2 · p. 3
  • The illustrated MLE-bench workflow iterates idea generation, implementation and experimentation, result analysis, and submission-based feedback into new ideas.The paper focuses on the idea-generation step within this cycle.Source: §2 · p. 4

3 Methods

The methods combine trajectory analysis on MLE-bench with a controlled prompt intervention on MLE-bench lite. Ideation diversity is measured from planned model architectures, while agent scaffolds and LLM backbones define the systems compared.

3.1.1 Dataset and 3.1.2 Metrics

  • Trajectory analysis uses MLE-bench, while the controlled experiment uses its curated 22-task MLE-bench lite subset; performance is measured as medal rate.Medal rate is the percentage of attempts earning bronze, silver, or gold under task-specific percentile thresholds.Source: §3.1.1 · p. 4, §3.1.2 · p. 4

3.1.3 Agentic Scaffolds

  • The study compares AIDE, AIRA-Greedy, and AIRA-MCTS scaffolds, whose search policies, operators, and memory configurations shape candidate-solution trees.Draft creates initial solutions, Debug corrects errors, and Improve seeks better-performing solutions.Source: §3.1.3 · p. 4

3.1.4 LLM Backbones

  • Trajectory analysis uses o3, two gpt-oss sizes, Llama Maverick, Devstral, and CWM, while the controlled experiment uses DeepSeek R1.The studied backbones span different model sizes and architectures.Source: §3.1.4 · p. 5

3.2 Measuring Ideation Diversity

  • The diversity measure focuses on agents’ planned model architectures, using base-2 Shannon entropy over their architecture distribution.Agents generate at most five initial ideas, from which the study extracts high-level approaches and specific models; diversity in other engineering aspects is outside scope.Source: §3.2 · p. 5

3.3 Controlling Ideation Diversity

  • The controlled intervention changes the system prompt to compare baseline agents with agents whose ideation diversity is ablated.The authors intend the prompt changes to affect idea diversity rather than other solution aspects such as implementation quality.Source: §3.3 · p. 5, §3.3.2 · p. 5

3.3.1 Baseline agents

  • Baseline agents use sibling memory, prompt-adaptive complexity, and explicit system-prompt instructions to encourage different solutions.Complexity cues progress from minimal to moderate to advanced across the five initial ideas.Source: §3.3.1 · p. 5

3.3.2 Agents with ablated diversity

  • The ablated agents remove prompt-adaptive complexity and diversity instructions, while reusing sibling memory to request similar ideas.This setup aims to reduce diversity while retaining the sibling-memory mechanism.Source: §3.3.2 · p. 5

4 The Ideation Diversity Bottleneck

The section analyzes how agent scaffolds shape ideation diversity, tests diversity reduction through prompting, and checks whether the performance pattern holds across additional metrics.

4.1 Deep-dive on Agentic Trajectories

  • AIDE concentrated 70% of initial draft ideas in GBDT and CNN, while AIRAGreedy’s four most common architectures accounted for 68% (Figure 3).For intended models, LightGBM and EfficientNet made up 43% for AIDE, while nine models represented that share for AIRAGreedy.Source: §4.1 · p. 6, Figure 3 · p. 6
  • Higher-performing backbones used 3.5 distinct architectures on average in their first five ideas, versus 2.8 for the other studied backbones, and tree-level diversity correlated with performance (Figure 4).Tree-level diversity is the average number of architectures used in the first five exploration nodes.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2.1 Are We Actually Influencing Diversity?

  • Prompting agents to generate similar ideas increased the share of tasks using no more than two distinct architectures or approaches in the first drafts from 40% to 70% (Figure 5).The comparison is between baseline AIRAGreedy and AIRAMCTS agents and their low-diversity variants.Source: Figure 5 · p. 7, §4.2.1 · p. 8

4.2.2 Results

  • Reducing ideation diversity lowered MLE-bench lite medal rate by 6.9 absolute points for AIRAGreedy and 8.4 for AIRAMCTS (Figure 6).The reduction was induced by prompting agents to generate similar ideas.Source: §4.2.2 · p. 8, Figure 6 · p. 8

4.3.1 Alternative Metrics

  • The paper evaluates valid submission rate, average normalized score, percentile, and ELO-based agent ranking alongside medal rate.These metrics capture different aspects of performance, including submission ability, scores relative to human distributions, and head-to-head rankings.Source: §4.3.1 · pp. 8–9

4.3.2 Data Analysis Results with Additional Metrics

  • Diversity’s correlation with performance remained consistent and was higher when performance was measured by average normalized score or percentile rather than medal rate (Figures 7 and 8).Source: §4.3.2 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9

4.3.3 Controlled Experiment Results with Additional Metrics

  • Across all additional metrics, diversity-ablated agents lose performance versus baselines (Figure 9), supporting poorer performance when ideas are less diverse.The paper also reports that valid-submission rates fell from 98% to 92% for AIRAGreedy and to 90% for AIRAMCTS, with repeated failed T5 attempts contributing to the decline.Source: §4.3.3 · pp. 9–10, Figure 9 · p. 9

5 Discussion

The discussion frames implementation quality as a current bottleneck and argues that ideation diversity can help agents avoid implementation pitfalls. The findings are limited to MLE-bench, and the authors note both evaluation and experimental-design caveats.

Ideation diversity

  • Diverse ideas can reduce implementation risk by helping agents design solutions they can execute, while the separate benefit of exploring different paths remains difficult to evaluate.A good plan can still fail when the agent cannot implement it; repeating the experiments as coding capabilities improve may clarify the issue.Source: §5 · p. 10

Implementation bottleneck

  • Agents that spend more time and a greater share of their task budget on successfully implemented solutions tend to earn more medals, suggesting implementation capacity constrains performance.The authors hypothesize that planning’s relative importance may grow as coding improves, shifting its role from de-risking implementation toward exploring the solution space.Source: §5 · p. 10

Generalization to other benchmarks

  • The study’s findings come from MLE-bench; generalization to other machine-learning tasks is a hypothesis, and additional benchmarks remain future work.The authors point to the benchmark’s range of tasks as a reason generalization may be likely, not as evidence that it has been established.Source: §5 · p. 10

Limitations of MLE-bench evaluations

  • Medal outcomes are not consistent across competitions because thresholds depend on submission counts; the bronze-to-best-score gap is often below 3%.Evaluation also faces variability from differences between agents’ custom test sets and the Kaggle private test sets used to compute thresholds.Source: §5 · p. 10

Limitations of this study

  • The diversity intervention may have second-order effects that are difficult to track, and the authors suggest separating the ideating and implementing LLMs to better isolate its effect.The paper also experimented with controlling diversity through temperature.Source: §5 · p. 11

6 Related Work

Related work connects this study to research on generation and trajectory diversity, multi-agent systems, and automated machine learning. It also situates MLE-bench and AIRA among benchmarks and agent frameworks for machine-learning tasks.

Generation diversity in language models

  • Prior sequence-generation work studied selecting or combining diverse yet plausible outputs, with later work examining diversity in neural machine translation and other sequence-to-sequence settings.Related work also includes nucleus sampling for coherent and diverse text and studies of generation diversity in LLM training and alignment.Source: §6 · p. 11

Diversity in reinforcement learning and population-based reinforcement learning

  • Reinforcement-learning research uses trajectory diversity and novelty objectives to support exploration, learn skills, or diversify populations and policies.The cited work includes diversity-aware policy optimization and B-star, which balances exploration and exploitation for reasoning LLMs.Source: §6 · p. 11

Diversity in multi-agent systems

  • Multi-agent research studies behavioral diversity, its effects on exploration and training efficiency, and prompt-based control of conversational diversity.Examples include CTEM’s trajectory entropy maximization and SemDiv’s use of natural-language coordination strategies to shape teammate policies.Source: §6 · p. 12

Automated machine learning and AI research agents

  • MLE-bench comprises 75 Kaggle machine-learning tasks, while AIRA formalizes AI research as search policies over candidate solutions.The cited landscape also includes agents evaluated on machine-learning tasks and a self-contained task focused on language models.Source: §6 · p. 12

7 Conclusions

The authors conclude that ideation diversity is a key bottleneck in AI research-agent performance, supported by trajectory analysis and a controlled experiment. They report that the findings hold across several evaluation metrics and recommend diversity-aware methods, multiple metrics, and newer benchmark tasks.

7 Conclusions

  • Trajectory analysis and a controlled experiment support the conclusion that ideation diversity is a key bottleneck in AI research-agent performance.The authors report that the findings hold across several evaluation metrics.Source: §7 · p. 12
  • The authors recommend focusing on diversity-aware methods as implementation quality becomes less important when AI systems improve.This is presented as a future-work direction tied to continued improvement in AI systems.Source: §7 · p. 12
  • They recommend considering multiple evaluation metrics and extending existing benchmarks to more recent machine-learning tasks.Source: §7 · p. 12

A.1 Controlling Ideation Diversity Using Temperature

The authors test whether sampling temperature can control ideation diversity in AIRAGreedy after removing other diversity-enhancing mechanisms. Temperature changes do not affect medal rate, while Elo is the exception among the alternative metrics; the authors caution that temperature may affect other agent capabilities too.

A.1 Controlling Ideation Diversity Using Temperature

  • The experiment uses AIRAGreedy with sibling memory, prompt-adaptive complexity, and system-prompt diversity language removed, then varies temperature around DeepSeek-R1’s recommended 0.6.Source: §A.1 · p. 17
  • The authors leave further investigation of temperature-based results to future work because changing temperature may have additional effects that are hard to reason about.They give potential effects on implementation capabilities as an example.Source: §A.1 · p. 17
  • Changing temperature does not affect medal rate, while higher temperature significantly improves Elo; valid submission rate, average normalized score, and percentile show no such impact.The authors hypothesize that temperature changes may affect the agent in other ways, including implementation capabilities.Source: Figure 12 · p. 17, §A.1 · p. 17

A.2.1 Limitations of MLE-Bench Medal System

MLE-bench’s medal system has threshold and comparability limitations, motivating additional evaluation metrics. The appendix also reports that AIDE uses a narrower range of image-classification architectures than AIRAGreedy.

A.2.1 Limitations of MLE-Bench Medal System

  • Bronze thresholds range from the top 10% to the top 40% across competitions, so medals do not represent the same performance level.Source: §A.2.1 · p. 18
  • In about 30% of MLE-bench competitions, the best score is less than 3% above the bronze threshold; the gap is similarly small in around 50% of MLE-bench lite competitions.Figure 13 shows the ratio between bronze thresholds and best scores; ratios close to 1 indicate narrow margins.Source: Figure 13 · p. 18, §A.2.1 · p. 18
  • Agents use custom test sets rather than Kaggle’s private sets, and older human-score distributions may not reflect current performance.The authors report that all tested agents fail to earn medals on most competitions held after 2022.Source: §A.2.1 · p. 18

A.2.2 Evaluation Principles

  • No additional metric satisfies all evaluation principles, while using too many metrics can make agent assessment and comparison difficult.Table 2 relates the additional metrics used in the work to those principles.Source: Table 2 · p. 19, §A.2.2 · p. 19

A.3 Models Used By Agents For Image Classification Tasks

  • In image classification, AIDE uses EfficientNet variants for almost 40% of tasks and only three architectures in 75% of attempts; AIRAGreedy’s EfficientNet, ConvNeXt, and ViT ideas make up 38% of initial ideas.Source: Figure 14 · p. 19, §A.3 · p. 19

Related papers

Loading 2511.15593v2…