AIM: Agentic Idea Management for Automated Research

1. Introduction

• Automated research must allocate limited experimental resources across candidate approaches while supporting both discovery of directions and refinement of implementations.Implementation and evaluation are costly.Source: §1 · p. 1
• Idea-driven research faces three central challenges: organizing ideas, selecting which to investigate, and ensuring implementations align with their intended ideas.AIM is introduced to address these challenges.Source: §1 · pp. 1–2
• Solution-driven methods optimize executable solutions, whereas idea-driven methods reason over research directions before delegating implementation to a solver.The paper says idea-driven methods facilitate comparison across approaches, lesson transfer, and interpretation of research trajectories.Source: §1 · p. 1, Figure 1 · p. 2
• AIM’s Agentic Surrogate organizes idea clusters, its Acquisition mechanism selects ideas, its Auditor checks validity and idea–solution integrity, and its Resource Planner adjusts parallelism under a fixed budget.The selected ideas are implemented and evaluated by parallel solvers, and audited evidence guides subsequent research directions.Source: §1 · p. 2
• AIM scores 67.0% on System Optimization and 55.8% on long-horizon Model Development & CUDA, exceeding ScientistOne by 1.6 and 4.9 percentage points, respectively.AIM reaches ScientistOne’s best score up to 3.1× faster.Source: §1 · p. 2

Solution-driven Search

• Solution-driven methods search executable artifacts: AIDE [Jiang et al., 2025] and AIRA [Toledo et al., 2026] use tree search, while AlphaEvolve [Novikov et al., 2025] uses evolutionary mechanisms.Tight evaluator–solver coupling lets feedback guide implementation changes, but can entangle research progress with artifact-specific engineering decisions.Source: §2 · p. 3

Idea-driven Search

• Idea-driven systems represent ideas or hypotheses explicitly; AI Scientist-v2 [Yamada et al., 2025], MARS [Chen et al., 2026b], and Arbor [Jin et al., 2026] manage trees, while DeepScientist maintains a list.ScientistOne [Meng et al., 2026] uses a beam-search-like algorithm to retain the best-performing ideas.Source: §2 · p. 3
• Idea-driven search must organize growing idea pools and evidence, choose ideas and budget allocations, and verify that solver outputs faithfully implement selected ideas.Prior methods use fixed rules such as UCB or MCTS [Yamada et al., 2025], or LLM judgments without explicit estimates [Jin et al., 2026].Source: §2 · p. 3

3. Automated Research: Problem Definition

• Evaluating every plausible idea is generally infeasible because implementation and verifier calls are expensive, so search needs structured representations and principled budget-aware navigation.The dominant cost is obtaining reliable quality evidence through implementation and verification, which can be expensive and noisy.Source: §3 · pp. 3–4

Idea and Solution Spaces

• An idea is a natural-language candidate approach; an implementation process maps it to an executable solution that is scored by a fixed verifier.The agent has access only to a finite pool of ideas discovered so far.Source: §3 · pp. 3–4

Objective

• The objective is to identify the highest-performing idea within a computational budget measured in experiment executions and/or runtime hours.The agent must determine which subset of ideas to explore first.Source: §3 · p. 4

4. The AIM Framework

• AIM combines an Agentic Surrogate, Agentic Acquisition, independent Solvers, a Solution Auditor, and a Resource Planner in an iterative idea-search framework.Audited observations and lessons update the search state and expand the idea pool for subsequent iterations.Source: §4 · p. 4

4.1. Agentic Surrogate: Organize and Estimate

• The Agentic Surrogate organizes related ideas into evolving semantic clusters and estimates their relative promise using experimental evidence.It uses ordinal estimates at both cluster and idea levels, considering observed performance, evidence scarcity, semantic novelty, and implementation lessons.Source: §4.1 · pp. 4–5

4.2. Agentic Acquisition: Dispatch, Solve, and Expand

• The Agentic Acquisition balances exploration and exploitation at cluster and idea levels, dispatches ideas to independent Solvers, and expands the pool using audited results.Expansion updates implementation memory and generates candidates through score-guided refinement, cross-pollination, error-guided repair, or novel idea generation; novel ideas broaden the search space.Source: §4.2 · pp. 5–7

4.3. Solution Auditor

• The Solution Auditor checks whether implementations match their intended ideas, meet task requirements, and obtain scores without exploiting the verifier.It discards trivial, task-mismatched, or reward-hacked results; for idea mismatches, it reconstructs the input idea so history and lessons reflect the evaluated mechanism.Source: §4.3 · p. 7

4.4. Resource Planner

• The Resource Planner distributes a fixed total of Solver branches across iterations, trading parallel breadth against more frequent feedback-driven updates.It preserves total branch and execution budgets and determines when the pipeline terminates, after which the best-scoring solution is selected.Source: §4.4 · p. 7

5. Experiments

• Evaluation covers ten AutoLab System Optimization, Model Development, and CUDA tasks, comparing solution-driven and idea-driven baselines under task-specific experiment and time limits.The benchmark spans five system-optimization tasks, two model-development tasks, and three CUDA tasks.Source: §5 · p. 8

5.1. Results

• AIM averages 67.0% on System Optimization, 1.6 points above ScientistOne (65.4%) and 3.7 above AdaEvolve (63.3%).On Flash Attention, its gains over ScientistOne and AdaEvolve are 4.3 and 5.2 points, respectively.Source: §5.1 · pp. 8–9
• On long-horizon tasks, AIM achieves the highest mean score on three of five, improving over ScientistOne by 4.2 points on Moving MNIST, 0.4 on Huffman, and 2.1 on NTT, but trails AdaEvolve on Data Selection IFEval.AIM scores 61.8 on Data Selection IFEval, below AdaEvolve’s 82.7.Source: Table 2 · p. 9, §5.1 · p. 9
• On Flash Attention, AIM reaches competitors’ final score levels in 1–2 hours versus their 2.2–5.5 hours, then reaches 90.5% after 3.3 hours.AdaEvolve reaches 85.3% at 5.5 hours, while ScientistOne reaches 86.2% at 3.5 hours.Source: §5.1 · p. 9, Figure 4 · p. 10

5.2. Ablation Studies

• On Flash Attention, removing any major component lowers the full model’s 90.5% score; removing the Surrogate causes the largest drop, 4.6 points.The drops are 2.9 points without the Auditor, 2.4 with a fixed resource schedule, and 1.3 without explicit Acquisition actions; ablations alter idea organization, selection, auditing, or allocation.Source: §5.2 · pp. 9–10, Figure 5 · p. 10

5.3. Qualitative Examples

• In Data Select IFEval iteration 3, AIM groups 33 ideas into four clusters; an explored low-ranked idea matches the 0.377 best score while an exploited pick regresses to 0.119.The Estimator ranks “metadata stratification” first based on its member idea’s performance and demotes generative-probing approaches based on earlier failures.Source: §5.3 · p. 10

6. When to Search Over Ideas? A Formal Understanding

• Semantic coverage counts distinct research directions represented by evaluated solutions; multiple solutions can map to the same direction and count as one.The mapping π groups solutions by underlying idea, so a trajectory through four solutions may cover only two semantic directions.Source: §6 · pp. 11–12
• For at most N evaluated executable solutions, semantic coverage C cannot exceed C_idea when an extreme idea-driven procedure assigns N executions to N distinct ideas.This is Proposition 1’s bound on attainable semantic coverage.Source: §6 · p. 12
• Idea-driven search makes semantic breadth explicit and controllable, but solution-driven search can attain the same bound if its code-level search produces semantically distinct solutions.The advantage depends on the task: it is greater when many directions exist but few are competitive, and limited when K/G_ε is small.Source: §6 · pp. 12–13
• C ≥ (K/G_ε) ln(1/δ) is sufficient to discover an ε-optimal direction with probability at least 1 − δ, when feasible.Figure 6 visualizes the requirement for different target success probabilities.Source: §6 · pp. 12–13, Figure 6 · p. 13

6.1. Empirical Support

• Idea-driven methods explore substantially broader regions on Flash Attention, while the convex-hull-area gap between paradigms is considerably smaller on Radix Sort.The analysis embeds intermediate solutions, projects them with PCA, and uses convex-hull area as a rough proxy for explored breadth.Source: §6.1 · p. 14, Figure 7 · p. 14

6.2. Practical Considerations

• The coverage analysis assumes faithful realization, but implementations may be incomplete, incorrect, or require tuning and low-level choices before a direction’s value is clear.Machine-learning systems may require hyperparameter tuning, while systems optimization may depend on implementation choices not specified at the idea level.Source: §6.2 · p. 14
• Allocating every experiment to a new idea maximizes breadth but may under-invest in optimization; repeatedly refining one implementation uses execution feedback but leaves less budget for alternatives.The appropriate balance depends on whether a task offers many meaningfully different high-level approaches or mostly implementation-level optimization.Source: §6.2 · p. 14
• AIM combines explicit idea-level search with iterative solver refinement and audits whether implementations remain aligned with selected ideas.Future work could adapt the exploration–refinement balance to implementation difficulty, solver fidelity, and task semantics.Source: §6.2 · p. 15

7. Conclusion

• AIM dynamically organizes an evolving idea pool, estimates research-direction promise, balances exploration and exploitation, and audits implementations before using their outcomes in later search.Source: §7 · p. 15
• Across 10 AutoLab tasks, AIM achieves the best overall performance and reaches strong solutions substantially faster than baselines.Source: §7 · p. 15
• The analysis concludes that explicit idea-level allocation makes semantic coverage controllable and broader coverage matters more when competitive directions are sparse among many plausible alternatives.Source: §7 · p. 15

AI Use Statement

• Generative AI tools were used to polish writing and generate plots.Source: AI Use Statement · p. 15
• The tools were not used to run the study’s experiments outside the agent methods being evaluated.Source: AI Use Statement · p. 15
• The authors reviewed AI-assisted work and take responsibility for the final content, including text, claims, and artifacts.Source: AI Use Statement · p. 15

Ethics Statement

• The work involves no human subjects, personal data, or user-facing deployment.Source: Ethics Statement · p. 15
• The authors identify risks including insecure implementations, verifier exploitation, and unintended task behavior, and use isolated, containerized benchmarks with fixed verifiers and a Solution Auditor as safeguards.Source: Ethics Statement · p. 15
• The safeguards cannot eliminate all risks, and human review remains necessary before consequential scientific or engineering use.Source: Ethics Statement · p. 15

• Algorithm 1 provides pseudocode for the complete AIM pipeline, with Appendix C describing the evaluated benchmarks and implementation details.Source: Reproducibility Statement · p. 16
• The appendix reports three independent runs using a common backbone model and evaluation environment, with means and standard errors.Source: Reproducibility Statement · p. 16

A. Related Works

• Earlier idea-driven systems use lists, trees, or beam-style retention with predefined selection mechanisms, whereas AIM autonomously reorganizes ideas and adjusts exploration or exploitation using evidence.Source: A. Related Works · pp. 20–21
• Automated-research testbeds include MLAgentBench, MLE-bench, RE-Bench, MLGym, MLRC-Bench, MLR-Bench, and ResearchGym; AIM primarily evaluates on AutoLab.Source: A. Related Works · p. 21
• AIM builds on prior work by distinguishing solution-driven search from idea-driven search and focusing on evolving idea representations, evidence-grounded selection, and idea–solution integrity.Source: A. Related Works · p. 21

B. Agentic Idea Manager Algorithm

• Algorithm 1 takes task context, an initial idea pool, an execution budget, and branch counts; a shown iteration adds ideas and advances the branch and iteration counters.Source: B. Agentic Idea Manager Algorithm · p. 22

C.1. Further Baseline and Benchmark Details

• Each method uses gemini-3.1-pro-preview [Team et al., 2023], task-specific execution and wall-clock limits, at most five parallel workers, and three runs per method-task pair.The limits are 300 executions and 6 hours for system optimization, 60 and 24 hours for model development, and 60 and 12 hours for CUDA tasks; idea-driven baselines receive the same brief, and results report mean and standard error.Source: §C.1 · p. 22
• The benchmark covers ten AutoLab [Xu et al., 2026] tasks across CPU system optimization, GPU model development, and CUDA kernels, each with a weak editable baseline and local evaluator.Agents can edit, execute, inspect scores and correctness feedback, and refine solutions; human-written references calibrate scoring but are not exposed to agents.Source: §C.1 · pp. 22–23
• Solutions failing correctness or task constraints, or not improving on the baseline, receive zero reward; valid runtime and model-development solutions use category-specific scoring scales.Runtime tasks compare generated solutions with baseline and reference runtimes, while model-development scores interpolate between benchmark-provided baseline and reference anchors.Source: §C.1 · p. 23

C.2. Implementation Details

• AIM starts with five parallel Solver branches and then plans 3–10 branches per round, redistributing a fixed branch set while keeping each branch’s execution budget fixed.For example, 300 executions divided among 25 total branches gives each branch 12 attempts, including executions later rejected by the Solution Auditor.Source: §C.2 · pp. 23–24
• AIM seeds its pool with approximately ten feasible ideas from ScientistOne’s Ideator [Meng et al., 2026], clusters ideas into 2–5 groups, and caps additions at three ideas per branch per round.The feasibility critic filters impractical ideas, while Organize chooses cluster count, descriptions, and assignments within the fixed bounds.Source: §C.2 · p. 24

C.3. Prompt Templates

• The surrogate prompts organize ideas into thematic clusters and rank clusters and ideas using observed scores, evaluation counts, theme quality, and mechanism specificity.Cluster ranks consider promise and exhaustion; idea ranks favor concrete, testable mechanisms aligned with the cluster theme.Source: §C.3 · pp. 24–26
• The acquisition prompts assign branches to clusters and distinguish exploiting productive directions from exploring underprobed clusters or novel ideas.For idea-level exploration, the selector prefers higher-ranked unevaluated ideas with untried mechanisms; exploitation favors the lowest within-cluster rank and can use successful parent ideas as evidence.Source: §C.3 · pp. 27–28
• The templates bootstrap with each branch’s highest-scored idea, then use swarm context to refine ideas, audit solutions, and allocate a fixed branch budget.The auditor checks for reward hacking, idea-solution mismatch, task mismatch, and trivial solutions; the planner pins Iter-1 and requires spending the exact remaining branch budget within validated per-iteration limits.Source: §C.3 · pp. 29, 31, 32

D.1. Solver Substitution with Claude Code

• With Claude Code, AIM consistently outperforms or matches ScientistOne across all five AutoLab benchmarks.Table 3 summarizes the comparison across the benchmark tasks.Source: §D.1 · p. 33, Table 3 · p. 33

D.2. Further Ablation Studies on the Agentic Surrogate

• 90.5 ± 0.8 is Full AIM’s Flash Attention score; removing Organize, Estimate, or the entire surrogate yields 89.0±0.4, 87.9±1.2, and 85.9 ± 2.7, respectively.The ablation results show that both Organize and Estimate contribute to performance.Source: Table 4 · p. 34, §D.2 · p. 34
• 83.4 ± 2.8 for Embedding-based Organize is the lowest score, while Direct Score Estimation scores 89.3±1.2, below Full AIM.The authors attribute these results to generic embeddings missing task-specific structure and ordinal comparisons being more reliable than raw score prediction.Source: §D.2 · p. 34

D.3. Time Efficiency Plots for All Tasks

• AIM reaches top scores in less wall-clock time than idea-driven and solution-driven baselines across the evaluated tasks.Figures 8 and 9 show time-to-score plots across all 10 tasks.Source: §D.3 · p. 35, Figure 8 · p. 35, Figure 9 · p. 36
• AIM generally uses more executions to reach its best score, which the authors conjecture reflects its early exploration of multiple directions.The execution-to-score comparisons allow five parallel workers for parallel baselines, with task budgets of 6 hours for System Optimization, 24 hours for Model Development, and 12 hours for CUDA tasks.Source: §D.3 · p. 37, Figure 10 · p. 37, Figure 11 · p. 38

D.4. Token Cost Analysis

• In 9 out of 10 tasks, AIM uses fewer tokens than ScientistOne to reach its best score, reducing token cost up to 3.4× on Data Select IFEval.AIM generally requires a larger token budget overall, reflecting a trade-off between token cost and final performance.Source: §D.4 · p. 38, Figure 12 · p. 40

E.1. More Qualitative Examples

• Qualitative examples show idea clusters and lineage informing actions that exploit promising directions while probing alternatives.In one Flash Attention example, the method selects a rank-2 idea because its parent achieved the highest overall score, while also choosing a lower-confidence idea to explore; the iteration’s outcomes include a new best score of 0.889.Source: §E.1 · p. 41

E.2. Qualitative Mechanisms of the Organize Operator

• The qualitative analysis examines how Organize restructures the idea pool as candidates and experimental evidence accrue, using cluster best-score heatmaps to show exploration across iterations.Source: §E.2 · p. 43
• Flash Attention’s leading direction stabilizes around SIMD and instruction-level parallelism, whereas FFT Rust’s leading direction shifts among four explored clusters; cluster counts and breadth vary by task.Flash Attention proposed eight cluster themes, seven of which were explored; cluster membership could be regrouped between iterations. Figure 14 shows that these revisions preserve coherent trajectories while refining broad directions into specialized strategies.Source: §E.2 · p. 43, Figure 13 · p. 44, Figure 14 · p. 45

E.3. Examining the Preciseness of the Estimate Operator

• 0.56–0.91 mean cluster-wise Spearman correlations, averaging approximately 0.71 overall, show that ordinal estimates positively track observed performance across five tasks.Agreement was strongest on Z-order Range Scan and Flash Attention; it generally improved over iterations, though AES128-CTR and FFT trajectories were non-monotonic, possibly due to similarly promising clusters or implementation noise.Source: Figure 15 · p. 46, §E.3 · p. 46, Figure 16 · p. 47

E.4. Dispatch Operator Action Analysis

• Early iterations favor exploration across clusters and ideas, while exploitation grows as evidence accumulates, with task-dependent action distributions indicating an adaptive policy.Dispatch explicitly chooses explore or exploit both across clusters and within the selected cluster, conditioning those choices on the idea map and promisingness estimates.Source: §E.4 · pp. 46–47, Figure 17 · p. 48

E.5. Mode Distribution of the Expand Operator

• Across tasks, cross-pollination generates 35–41% of ideas, novel generation 29–32%, and score-guided refinement 24–31%, while error-guided repair remains a small fraction.The three modes other than error repair have similar usage rates; together, the modes combine refinement and recombination with new directions rather than collapsing to one strategy.Source: Figure 18 · p. 49, §E.5 · p. 49

E.6. Effect of Solution Auditor Idea Reconstruction

• Across tasks, idea reconstruction greatly reduces idea-mismatch flags; in one example, it prevents the allocator from treating an untested claim as the run’s best-supported finding.The solution used a custom AVX-512 minimax exponential despite the assigned idea proposing standard exp() to avoid FMA-port saturation, so the auditor reconstructed the idea to match what was actually implemented.Source: Figure 19 · p. 50, §E.6 · p. 50, Example 1: the code relied on the… · pp. 51–52

Example 2: the secondary component survived, the central claim did not

• In a second Flash Attention example, the auditor flags a mismatch because the code retains minimax exp() but uses key-sequence vectorization and horizontal reductions, contradicting the idea’s core sequence-dimension strategy.The proposed strategy would process 16 queries vertically to eliminate horizontal reductions; the code instead processes one query against 64 keys at a time.Source: Example 2: the secondary component… · pp. 52–53
• Reconstruction shows the rewarded kernel still performs horizontal sums, correcting attribution that would otherwise credit eliminating them; the described kernel vectorizes QK^T over keys and PV over features.The passage warns that attribution errors would teach the surrogate and allocator the wrong lessons about which mechanisms work, especially for high-scoring branches.Source: Example 2: the secondary component… · p. 54

F.1. A Note on the Finiteness of Idea Space X

• For a fixed task, a finite token vocabulary and a finite maximum idea-description length imply that the idea space is finite.The analysis defines semantic coverage over this finite idea space so comparisons across methods are meaningful.Source: §F.1 · p. 54

F.2. Proof of Propositions

• Broader semantic coverage is increasingly necessary when competitive directions are sparse among many admissible alternatives; sufficient coverage scales with K/Gε.For a fixed number of admissible directions, sparser competitive directions require broader coverage.Source: §F.2 · pp. 56–57

F.3. Convex Hull Area Box-and-Whisker Plots

• Figure 20 presents supplementary embedding-space analysis in which idea-driven approaches generally show broader solution coverage.The figure reports convex-hull area distributions; the caption points to cosine-similarity analysis in Appendix F.5.Source: Figure 20 · p. 57

F.4. Convex Hull Visualization

• Figure 21 provides visual examples of the convex hulls described in Section 6.Source: §F.4 · p. 58, Figure 21 · p. 58

F.5. Supplementary Cosine-Similarity-based Diversity Analysis

• The analysis embeds generated candidates and averages pairwise cosine distance across each pool, with results averaged over three runs per method-task.Source: §F.5 · pp. 58–59
• On Flash Attention, EvoX has D = 0.0153, roughly 7.5× narrower than AIM.Source: §F.5 · p. 59
• Solution-driven methods generally explore narrower areas than idea-driven methods, and the gap is smaller on Radix Sort than Flash Attention.Source: §F.5 · p. 59
• On Flash Attention, AIM and ScientistOne have D ≈0.115, roughly 3–4× the value produced by every solution-driven baseline.The table reports measured values of (47), with higher values indicating greater diversity.Source: Table 6 · p. 59, §F.5 · p. 59