R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science

1 INTRODUCTION

• MLE remains labor-intensive: even top Kaggle teams spend considerable time on trial-and-error, feature crafting, and hyperparameter tuning.The paper notes that state-of-the-art LLMs on MLE-bench achieve only a small fraction of human expert performance.Source: §1 · p. 2
• R&D-Agent separates MLE into research and development phases spanning six components, with simple LLM-first baselines for each aspect.Research covers planning, exploration paths, memory, and reasoning; development covers coding workflows and evaluation strategy.Source: §1 · pp. 2–3
• Existing agents often optimize only parts of the workflow, entangle steps, and leave alternatives unexplored, making improvements difficult to generalize or reuse.The framework supports systematic exploration, replaceable components, and plug-and-play alternatives.Source: §1 · p. 3
• The human-guided R&D-Agent configuration delivers significant gains on MLE-Bench and achieves new state-of-the-art results.The authors also report systematic ablations across the framework’s phases to isolate component contributions.Source: §1 · p. 3

2 PREREQUISITES

• An MLE agent seeks to maximize a task’s evaluation score under a limited time budget, iterating on development data without access to the final evaluation metric.The final score is measured on the unseen test set; a development-set proxy guides exploration.Source: §2 · p. 3
• The constrained objective selects a feasible solution while limiting its total generation time.The feasible-solution set and generation-time function define the constraint.Source: §2 · p. 3

3.1 FRAMEWORK OVERVIEW

• The framework divides work between four research components for discovering ideas and two development components for turning promising ideas into evaluated solutions.The research phase includes planning, exploration-path structuring, reasoning, and memory; development includes coding workflow and evaluation strategy.Source: §3.1 · pp. 4–5

3.2 FRAMEWORK FORMULATION

• The R&D loop repeatedly plans, selects exploration parents, retrieves context, generates an idea, develops and evaluates it, and updates an exploration graph until the time budget is met.The final solution is selected from the graph.Source: §3.2 · p. 5

3 R&D-AGENT

• A human-expert-guided configuration of the modular framework is reported to establish state-of-the-art performance on MLE-Bench.The authors describe this configuration as the result of navigating the framework’s design space.Source: §3 · p. 4, §3.3 · p. 5

3.3 A HUMAN-EXPERT-INSPIRED AGENT DESIGN

• Planning gradually increases computational budgets as promising directions emerge, while path structuring starts with diverse branches and then exploits strong candidates within each branch.Later planning can allow ensembles or cross-validation; the adaptive DAG prunes weaker paths and fuses results.Source: §3.3 · p. 6
• The reasoning pipeline analyzes the solution and dataset, forms and assesses hypotheses, and forwards a promising implementable idea; memory shares selected ideas across branches while preserving diversity.LLM-based assessment filters multiple generated ideas before development; memory sampling favors topical similarity, higher scores, and recency.Source: §3.3 · p. 6
• Coding uses rapid prototyping on a representative data subset before full evaluation, while standardized splits and aggregated validation support fair comparison across branches.Only runnable, logically sound solutions proceed to full-scale evaluation; top branch solutions are assessed on the same validation set.Source: §3.3 · p. 6

4.1 EXPERIMENT SETUP

• Experiments use a 12-hour budget and lower-throughput hardware than prior work, with comparisons averaged over three runs; ablations use a 40-competition subset.The subset includes competitions where R&D-Agent, AIDE, or ML-Master achieved medals; the research-phase ablation evaluates four components.Source: §4.1 · p. 7, §4.3 · p. 7

4.2 MAIN RESULTS

• 35.1% Any-Medal Rate on MLE-Bench exceeds ML-Master’s 29.3% by 5.8 percentage points; R&D-Agent also leads across task-complexity levels.The comparison uses the official benchmark metrics; Table 2 reports results as mean ± SEM across three runs.Source: §4.2 · p. 7, Table 2 · p. 8

4.3.1 RESEARCH PHASE ABLATION STUDY

• Without planning or reasoning, Any Medal Rate falls from 35.1% to 26.7%; without memory context, it falls to 32.0%.Removing reasoning also lowers Improve Rate from 41.1% to 23.0%; the memory ablation shows the smallest decline among these components.Source: §4.3.1 · pp. 8–9
• Removing adaptive exploration-path structuring causes the largest research-phase decline: Any Medal Rate falls to 25.3% and average loops from 45.9 to 19.4.The authors identify exploration-path structuring as the most critical advantage over existing approaches.Source: §4.3.1 · pp. 8–9

4.3.2 DEVELOPMENT PHASE ABLATION STUDY

• Removing the coding workflow drops Any Medal Rate to 24.0% immediately, while removing evaluation strategy yields delayed decline to 30.7%.The evaluation-strategy ablation initially matches the full system until hour 2; the development analysis tracks medal acquisition over 12 hours.Source: §4.3.2 · p. 9, Figure 3 · p. 9

6 CONCLUSION

• The framework separates research from development into six modular components, and the authors report that component ablations attribute gains to architecture rather than improved model capabilities.They also describe systematic architecture comparison and component-level attribution as practical uses of the framework.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT; ETHICS STATEMENT

• The paper reports that its complete codebase and component details are provided for reproducibility, and that experiments avoid sensitive or personally identifiable data.The authors also encourage domain-appropriate oversight for deployment.Source: REPRODUCIBILITY STATEMENT · p. 11, ETHICS STATEMENT · p. 11

A RELATED WORK

• Prior MLE agents often optimize only part of the workflow; R&D-Agent separates strategic research from tactical development and supports extensible design exploration.The related-work discussion describes MLE-Bench as a benchmark based on authentic Kaggle competitions.Source: A RELATED WORK · p. 14

C.1 PERFORMANCE ANALYSIS ACROSS DIFFERENT BACKEND LLMS

• 29.3% Any-Medal Rate for the hybrid configuration exceeds both single-model setups, including o3 only by 4.0 percentage points.Assigning reasoning models to research and code generators to development yields a 57% relative improvement over GPT-4.1 only.Source: §C.1 · p. 14, Figure 4 · p. 15
• Across the three backend configurations, performance ranks GPT-4.1 only below o3 only, with the hybrid configuration highest.The hybrid arrangement assigns reasoning models to research and code generators to development, selectively applying expensive reasoning models.Source: §C.1 · pp. 14–15

C.2 EFFECT OF EXTERNAL KNOWLEDGE

• The external knowledge base covers 85 other Kaggle competitions, and embedding similarity retrieves material during research for use in solution design.It contains notebooks and forum discussions spanning tabular data, computer vision, NLP, and time-series forecasting.Source: §C.2 · p. 15
• External knowledge harms overall performance, with particularly severe degradation on Low==Lite tasks; only high-difficulty competitions benefit.The authors hypothesize that retrieval may add noise on standard tasks where LLMs already know common patterns, and recommend applying RAG selectively by task complexity.Source: §C.2 · pp. 15–16

C.3 COMPUTATIONAL EFFICIENCY ANALYSIS

• R&D-Agent achieves state-of-the-art performance with one NVIDIA V100 GPU within 12 hours, using 2× less time and 8× fewer GPUs than the cited baselines.The comparison includes methods requiring multiple GPUs, such as MLE-STAR with 8×V100, or runtimes of 24 hours for most baselines.Source: §C.3 · p. 16, Table 5 · p. 16

C.4 COMPARISON WITH CLOSED-SOURCE SYSTEMS

• R&D-Agent approaches InternAgent’s 36.4% while surpassing Neo’s 34.2% using one-third the runtime, 12h versus 36h.The framework operates fully autonomously and completes the pipeline from problem analysis through implementation to evaluation.Source: §C.4 · p. 16, Table 6 · p. 16
• On Low==Lite tasks, R&D-Agent achieves 68.2%, outperforming both closed-source alternatives by substantial margins.The authors attribute this performance to the dual-phase design, modular components, and standardized evaluation protocols.Source: Table 6 · p. 16, §C.4 · p. 17

C.5 EXTENDED ANALYSIS: MLE-BENCH LITE RESULTS

• The Lite-subset comparison is included to support comparison with methods that report only Lite results.Figure 5 covers agents with reported results on this subset.Source: §C.5 · p. 17, Figure 5 · p. 17
• Figure 5 values are mean performance across benchmark tasks with SEM; AIRA values are zero because its original paper gave no explicit results.Source: Figure 5 · p. 17
• 68.2 ± 2.6% on MLE-Bench Lite is 4.6 percentage points above MLE-STAR with Gemini-2.5-pro at 63.6 ± 6.0%.The comparison spans early agents such as MLAB at 4.2 ± 1.5% and current state-of-the-art systems.Source: §C.5 · p. 17, Figure 5 · p. 17

D EXPERIMENTAL DETAILS AND SUPPLEMENTARY RESULTS

• Supplementary material includes detailed performance breakdowns and experimental design specifications.Source: D EXPERIMENTAL DETAILS AND… · p. 17

• Table 7 reports complete performance metrics for both R&D-Agent configurations.Source: §D.1 · p. 17
• The metrics cover three independent runs, making run-to-run variability visible.Source: §D.1 · p. 17
• The authors present the results to provide transparency into variability and statistical robustness.Source: §D.1 · p. 17

D.2 COST ANALYSIS

• At approximately $21 per competition, R&D-Agent achieves medal-winning performance.The reported cost is for GPT-5 main experiments across three independent runs.Source: §D.2 · p. 17, Table 8 · p. 18

D.3 MEMORY CONTEXT DESIGN

• Independent branches preserve diversity but can repeat exploration and miss knowledge transfer; the design enables controlled information exchange between them.A probabilistic interaction mechanism applies across candidate hypotheses.Source: §D.3 · p. 18

D.3.1 CANDIDATE HYPOTHESIS CONSTRUCTION

• The candidate pool combines current-branch hypotheses, globally best hypotheses, and hypotheses sampled from the interaction kernel.Source: §D.3.1 · pp. 18–19
• The interaction potential weights hypothesis similarity and score difference, with path-length decay reducing the hypothesis-information component over exploration.This makes score information more dominant in later stages.Source: §D.3.1 · p. 19

D.3.2 ADAPTIVE HYPOTHESIS SELECTION

• LLM selection can select, modify, or generate hypotheses, while its strategy adapts to exploration stage and remaining time.Source: §D.3.2 · p. 19
• The final multi-trace merge stage integrates complementary solutions discovered across independent explorations.Source: §D.3.2 · pp. 19–20

• The development-phase ablation studies use 40 MLE-Bench competitions.Source: §D.4 · p. 20
• The subset includes classification, regression, and time-series forecasting challenges.Source: §D.4 · p. 20
• The competitions span tabular, text, image, and multimodal data types.Source: §D.4 · p. 20

D.5 COMPARATIVE BASELINE MEDAL ACHIEVEMENT STATISTICS

• Across 75 Kaggle competitions, Table 9 summarizes medal frequencies for o3-4.1 and GPT-5 over three runs and reports significant variation in performance consistency.The table defines values as medals achieved per competition (0–3).Source: §D.5 · p. 20, Table 9 · p. 20

E.1 PLANNING

• Planning dynamically balances exploration breadth and depth by evaluating the exploration trace and remaining computational time rather than following a fixed schedule.Competition analysis extracts task, data, metric, and submission details to inform strategy and time management.Source: §E.1 · p. 23

E.2 EXPLORATION PATH STRUCTURING

• Exploration Path Structuring selects promising experiments using validation scores, generalizability, and overfitting risk, and merges successful components while avoiding known failure patterns.It coordinates parallel solution traces through branching, convergence, and diversity-aware selection.Source: §E.2 · pp. 23–24

E.3 REASONING PIPELINE

• The Reasoning Pipeline identifies a small set of high-impact competition challenges and formulates actionable, testable hypotheses evaluated across five dimensions.Challenge analysis covers data, models, evaluation, and implementation; hypotheses are scored for alignment, impact, novelty, feasibility, and risk-reward balance.Source: §E.3 · pp. 24–25

E.4 MEMORY CONTEXT

• Memory Context compares each experiment with prior best results, validates hypotheses, and transfers historical successes and failure lessons into later code generation.Its code-generation guidance emphasizes debuggability, reproducibility, error prevention, and resource management.Source: §E.4 · pp. 25–26

E.5 CODING WORKFLOW

• Coding Workflow uses debug-first iteration with sampled training data, minimal epochs, and timing estimates, then checks execution, competition compliance, and authentic predictions.The debug protocol samples 10% of training data, while evaluation also checks submission format and rejects constant, random, or hard-coded outputs.Source: §E.5 · pp. 26–27

E.6 EVALUATION STRATEGY

• When classes have too few samples for stratification, the split prioritizes representing every class in training before adjusting the validation split.The evaluation protocol also emphasizes class-balance awareness throughout assessment.Source: §E.6 · p. 28
• ValidationSelector re-evaluates top experiments from multiple exploration branches on a stratified 90-10 holdout, using parallel execution to compare candidates consistently.The framework describes this as a four-stage process: candidate collection, holdout generation, parallel re-evaluation, and performance ranking.Source: §E.6 · p. 28
• Candidates are ranked by holdout performance using standardized grading that accounts for whether higher or lower metric values are better.The grading protocol uses the competition-specific metric from the reference implementation and applies it consistently to the holdout test set.Source: §E.6 · p. 28

F CASE STUDY: COMPARATIVE ANALYSIS ON JIGSAW TOXIC COMMENT CLASSIFICATION

• Under identical 12-hour, GPT-5, single-V100 conditions, R&D-Agent earned bronze on Jigsaw while ML-Master performed slightly above median.The task is multi-label text classification with severe class imbalance.Source: F CASE STUDY: COMPARATIVE ANALYSIS ON… · p. 29
• R&D-Agent uses an asymmetric multi-label loss, adaptive batch sizing, a customized RoBERTa classifier, and head-plus-tail truncation, unlike ML-Master’s standard BCE loss and unmodified model.The truncation strategy retains text from both its beginning and end; the debug module is described as reducing development time.Source: F CASE STUDY: COMPARATIVE ANALYSIS ON… · p. 29

F.1 CODE OF R&D-AGENT

• The simplified R&D-Agent code reserves special tokens and, when text exceeds its content budget, keeps 75% of the head and 25% of the tail.The retained tokens are wrapped with special tokens and padded to the batch maximum length, bounded by max_length.Source: §F.1 · p. 30

F.2 CODE OF ML-MASTER

• The simplified ML-Master code uses BCEWithLogitsLoss and selects the best checkpoint by validation AUC during a two-epoch training loop.The code records the validation-selected model state for later evaluation.Source: §F.2 · p. 33