PaperlayerView exact revision on arXiv

Source-linked AI summary

Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents

Patrick Tser Jern Kon, Jiachen Liu, Qiuyi Ding, Yiming Qiu, Zhenning Yang, Yibo Huang, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, Ang Chen

arXiv:2502.16069v2cs.AIcs.LG

TL;DR

LLMs are increasingly used across scientific work, but rigorous experimentation remains difficult to automate. Curie embeds reliability, methodical control, and interpretability into an agent framework, and achieves a 3.4× improvement in correctly answering experimental questions compared to the strongest baseline tested.

  • Problem

    LLM approaches to scientific work often rely on ad-hoc prompts prone to hallucination, and their ability to support rigorous experimentation remains limited.

    Source: §1 · p. 1

  • Method

    Curie combines an Architect and Technician agents with an Experimental Rigor Engine for reliability, methodical control, and structured experimental documentation.

    Source: §1 · pp. 1–2

  • Results

    3.4× improvement in correctly answering experimental questions compared to the strongest baseline tested.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Curie’s benchmark evaluation demonstrates its capability to automate rigorous experimentation.

    Source: §6 · p. 9

    The authors caution that over-reliance on AI in scientific discovery raises bias concerns and advocate human oversight through hybrid human–AI collaboration.

    Source: Impact Statement · p. 10

Abstract

from arXiv · show

Scientific experimentation, a cornerstone of human progress, demands rigor in reliability, methodical control, and interpretability to yield meaningful results. Despite the growing capabilities of large language models (LLMs) in automating different aspects of the scientific process, automating rigorous experimentation remains a significant challenge. To address this gap, we propose Curie, an AI agent framework designed to embed rigor into the experimentation process through three key components: an intra-agent rigor module to enhance reliability, an inter-agent rigor module to maintain methodical control, and an experiment knowledge module to enhance interpretability. To evaluate Curie, we design a novel experimental benchmark composed of 46 questions across four computer science domains, derived from influential research papers, and widely adopted open-source projects. Compared to the strongest baseline tested, we achieve a 3.4$\times$ improvement in correctly answering experimental questions. Curie is open-sourced at https://github.com/Just-Curieous/Curie.

1. Introduction

Curie targets the gap between LLMs’ growing use in scientific work and their limited support for rigorous experimentation. It combines agent coordination and rigor mechanisms with a benchmark of computer-science experimental questions.

1. Introduction

  • Existing LLM approaches often use ad-hoc prompts prone to hallucination, while rigorous experimentation remains a largely unexplored capability.The passage distinguishes these approaches’ effectiveness in creative tasks such as literature review and brainstorming from their limitations in rigorous experimentation.Source: §1 · p. 1
  • Curie’s Experimental Rigor Engine combines intra-agent reliability checks, inter-agent methodical control, and structured documentation for interpretability.The modules validate plans and setups, manage task transitions and scheduling, and maintain documentation for collaboration in large-scale experiments.Source: §1 · p. 2
  • The benchmark contains 46 experimental tasks across computer-science domains, derived from influential research papers and widely adopted open-source projects.Curie is presented as supporting reproduction, extension, and critique of existing research through experimentation.Source: §1 · p. 2
  • 3.4× improvement in correctly answering experimental questions compared to the strongest baseline tested.The authors describe Curie as a promising step toward accelerating scientific research.Source: §1 · p. 2

2. Background

The background frames experimentation as an iterative cycle and rigor as a combination of methodical procedure, reliability, and interpretability. It contrasts this with prior AI research and benchmarks that give less attention to rigorous experimental work.

2.1. Science Experimentation

  • Scientific experimentation iterates through design, execution, and documentation and analysis, with insights informing later hypothesis refinement.Design includes identifying variables and methods; execution includes configuring environments and exploring search spaces.Source: §2.1 · p. 2

2.2. Rigor in Experimentation

  • Rigor comprises methodical procedure, reliability, and interpretability across the experimental process.The background connects reliable, reproducible stages with trustworthy findings and consistent documentation with replication, understanding, and extension.Source: §2.2 · pp. 2–3

2.3. Related Work

  • Prior AI-for-science work covers several research stages, but experimentation remains underexplored, and existing end-to-end agents lack systematic enforcement of rigor principles.The paper contrasts its experimentation benchmark with benchmarks focused on deterministic reasoning, problem-solving, retrieval, or machine-learning tasks.Source: §2.3 · p. 3

3. Curie: Rigorous Experimentation

Curie places an Experimental Rigor Engine between an Architect and Technician agents to validate work, manage coordination, and track experimental progress. Its modules address reliability, methodical control, and interpretability throughout the workflow.

3.1. Architectural Overview

  • Curie’s Architect plans experiments while Technician agents implement and execute them, with the Experimental Rigor Engine intercepting actions for validation and routing.The Experiment Knowledge Module stores and tracks progress in structured, metadata-enriched plans.Source: §3.1 · p. 4

3.2. Intra-Agent Rigor Module - Reliability

  • Intra-ARM validates agent tasks step by step, enabling early error detection and correction rather than relying only on end-to-end checks after an experiment.Its extensible validators include checks that technician setups align with the plan and execution checks in a clean environment, with repeated runs for consistency.Source: §3.2 · pp. 4–5

3.3. Inter-Agent Rigor Module - Methodical Control

  • Inter-ARM divides plans into partitions, enforces permissible transitions among agents, and schedules work using priorities, allowed transitions, and agent availability.Partitioning supports modular execution and parallelization, while the scheduler aims to improve resource efficiency without departing from methodical procedures.Source: §3.3 · pp. 5–6

3.4. Experiment Knowledge Module - Interpretability

  • The Experiment Knowledge Module records plan changes as a DAG-like history and validates role-limited writes to preserve structured, interpretable experimental records.The history captures hypotheses, variable changes, and decision rationales; technicians can append results to assigned partitions but cannot alter unrelated sections.Source: §3.4 · p. 6

4. Experimentation Benchmark

The Experimentation Benchmark treats research tasks as experimental processes rather than isolated problems with fixed solutions. It specifies task context and ground truth, and characterizes complexity across experiment-driven dimensions.

4. Experimentation Benchmark

  • The benchmark is designed to stress-test agents on real-world experimentation challenges and comprises 46 tasks across four computer-science domains.Tasks derive from influential research papers and use cases in popular open-source projects.Source: §4 · pp. 6–7, Table 1 · p. 7

4.1. Experiment-Centric Task Design

  • Tasks require hypothesis formation, iterative refinement, and rigorous validation rather than one-shot problem-solving.Each task frames an experiment question with practical constraints and setup requirements, supplies domain knowledge and starter tools, and evaluates experiment design and result analysis against ground truth.Source: §4.1 · p. 7

4.2. Experimental Complexity

  • Benchmark complexity spans experiment design, setup, variable–outcome relationships, and goals involving competing objectives or trade-offs.These dimensions range from choosing variables and configuring environments to handling non-monotonic relationships and multi-objective balancing under constraints.Source: §4.2 · pp. 7–8

5. Evaluation

Evaluation compares Curie with OpenHands and Magentic on a 46-task, four-domain benchmark, using five independent trials per task and a hybrid assessment protocol. Curie consistently outperforms the baselines across reported metric and domain breakdowns.

5. Evaluation

  • The benchmark spans 46 research tasks in four computer-science domains, with five trials per task; GPT-4o powers all agents, and evaluation combines LLM judging with manual alignment checks.Experts cross-check a subset of judge assessments against annotations.Source: §5 · p. 8

5.1. Benchmark Performance

  • Curie consistently outperforms both baselines across all four experimentation-performance metrics.The metrics assess experiment design, execution setup, implementation alignment, and conclusion correctness.Source: §5.1 · p. 8, §5 · p. 8
  • Execution setup performance rises from 66.7% to 92.7%, while Curie’s conclusion correctness reaches 36.1%, versus 10.5% for OpenHands and 2.3% for Magentic.The paper attributes execution reliability to Intra-ARM validation and Inter-ARM task transitions, and conclusion performance to structured experiment documentation.Source: §5.1 · pp. 8–9
  • Across four task domains, Curie consistently outperforms the baselines; its conclusion accuracy is 6.5× OpenHands’ on cloud-computing tasks and 7.3× higher on ML-training tasks.The ML-training tasks lack detailed environment setup instructions, and the paper says Curie uses reflection and refinement to find the correct setup.Source: §5.1 · p. 9
  • Performance generally declines as complexity increases, but the decline is sublinear, and the authors suggest making the hardest tasks more challenging.Figure 8 reports average scores across four evaluation metrics at varying difficulty levels and complexity dimensions.Source: §5.1 · p. 9, Figure 8 · p. 9

6. Conclusion and Future Work

The conclusion presents Curie’s rigor-focused framework and benchmark as a step toward automated experimentation, while identifying interdisciplinary adaptation and human oversight as continuing concerns. The impact statement describes potential applications and future directions rather than established cross-disciplinary results.

6. Conclusion and Future Work

  • Adapting Curie to interdisciplinary research remains an open challenge involving domain-specific methods, uncertainty control, and extended time scales such as long-term biological studies.The authors also identify knowledge reuse across experiments as a possible way to improve efficiency.Source: §6 · pp. 9–10

Impact Statement

  • Curie combines methodical control, execution reliability, and structured knowledge management, and its evaluation reports higher conclusion accuracy and execution reliability than state-of-the-art agents.The authors introduce a benchmark of research-level challenges across four computer-science domains.Source: §6 · p. 9, Impact Statement · p. 10
  • The authors describe possible applications beyond computer science, including materials science, physics, and biomedical research, where complex setups and iterative hypothesis testing matter.These are proposed areas of potential application, not reported evaluations in those fields.Source: Impact Statement · p. 10
  • The impact statement argues that structured documentation and interpretability make Curie’s experimental processes auditable and traceable, while warning that AI-driven discovery raises bias and oversight concerns.It advocates human-AI collaboration in which AI assists rather than replaces scientific judgment.Source: Impact Statement · p. 10
  • Curie is presented as a foundation for future self-improving agents that refine experimental methods through continual learning.The authors describe this as an avenue for future work.Source: Impact Statement · p. 10

A. Curie Benchmark Complexity Explanation

The passage directs readers to Table 3 for the paper’s complexity-level definitions.

A. Curie Benchmark Complexity Explanation

B. Case Studies for Curie

Curie presents two case studies extending investigations of reasoning-step length in LLMs. The studies examine how optimal reasoning steps relate to model type and task complexity.

B. Case Studies for Curie

  • The two case studies extend work on reasoning-step length in LLMs from Jin et al. (2024).The paper describes the studies as extensions of that work.Source: B. Case Studies for Curie · p. 13
  • Curie extends an investigation of reasoning-step length by comparing GPT-4o and GPT-4o-mini and examining whether their optimal step counts differ.The experiment uses control and experimental groups to study accuracy across reasoning-step counts.Source: B. Case Studies for Curie · p. 13
  • A second case study examines how task complexity relates to the optimal length of reasoning chains, using gsm8k and last letters datasets.It varies reasoning-step counts while holding the model and method constant.Source: B. Case Studies for Curie · p. 14

C. Extended Evaluation: Fine-grained Performance Breakdown by Individual Metrics

Curie outperforms OpenHands and Magentic across alignment, conclusion, and design scores as task complexity and difficulty increase. Scores generally decline with greater complexity, with conclusion scores most affected.

C. Extended Evaluation: Fine-grained Performance Breakdown by Individual Metrics

D. Benchmark Details.

This section details benchmark questions and tasks for empirical investigations across language-model reasoning, vector indexing, cloud computing, and applied prediction and classification. It defines Vectorization for measuring execution-speed improvements in image-data computations and BabyLM for evaluating small-scale language models on child-directed text.

D. Benchmark Details.

  • The language-model questions probe sample-count and temperature effects, reasoning-chain length and construction, model and task dependence, and the impact of incorrect steps.They also ask whether smaller models with more samples or longer reasoning can meet accuracy targets more cost-effectively.Source: D. Benchmark Details · p. 16
  • The vector-search questions vary HNSW parameters and parallelism, compare index configurations under recall or latency constraints, and examine IVF and synthetic-data behavior.Examples specify SIFT1M settings including k=10, M=32, and efConstruction=40, and test for the highest recall under 5 ms latency.Source: D. Benchmark Details · p. 17
  • The cloud-computing questions compare instance efficiency and performance under fixed or mixed workloads, including matched-size instances from different CPU architectures.Other prompts seek a least-cost instance meeting a 99th-percentile latency below 150 ms for 500 concurrent add-to-cart requests.Source: D. Benchmark Details · pp. 17–18
  • Applied machine-learning prompts cover house-price prediction, review and feedback sentiment classification, passenger-outcome prediction, and disease-progression forecasting.The prompts frame these as prediction or classification tasks, emphasizing prediction error, generalization, accurate sentiment, or early diagnosis and monitoring.Source: D. Benchmark Details · p. 18
  • Vectorization measures processing-speed improvements for vectorized image-data computations by optimizing env/train.py and reporting execution speed for each tested configuration.The task goal is to improve the given script’s execution speed.Source: D. Benchmark Details · p. 18
  • BabyLM evaluates small-scale language models on child-directed text through perplexity and aims to improve performance on the BabyLM Benchmark.The task focuses on low-resource NLP tasks.Source: D. Benchmark Details · p. 18

E.1. Experimenter System Prompt Template

The experimenter prompt directs agents to design and conduct reproducible experiments, analyze real results, and justify conclusions. It also specifies behavioral expectations and criteria for judging experimental work.

E.1. Experimenter System Prompt Template

  • The prompt requires testable hypotheses, defined experimental variables, and control and experimental groups for valid comparisons.Experiments should be grounded in real, accurate facts.Source: §E.1 · p. 19
  • Experiments must use reproducible, interpretable workflows that can run as one cohesive program, with smaller modular programs allowed.The prompt calls for the workflow to be callable as a single program.Source: §E.1 · pp. 19–20
  • Agents must collect and analyze data systematically, cover the necessary search space, and draw justified conclusions from observed results.Results should support or prompt refinement of the hypothesis.Source: §E.1 · p. 19
  • Results must come from conducted experiments rather than simulation or guessing.The prompt explicitly rules out simulated results and unverified assumptions.Source: §E.1 · pp. 19–20
  • Evaluation checks hypothesis correctness and iteration count, plus whether the setup is reproducible, usable, interpretable, and scientifically rigorous.The prompt lists hypothesis formation and experimental setup as judged criteria.Source: §E.1 · pp. 19–20
  • Evaluation also checks whether results were produced through experimentation and sufficiently justify conclusions, which must follow logically and cover the search space.The prompt separately lists results generation and conclusion derivation criteria.Source: §E.1 · p. 20
  • Agents should justify decisions clearly and deliver scrutinizable, reproducible workflows, naming the runnable program experimental_workflow.sh.The prompt frames rigor, accuracy, and interpretability as behavioral expectations.Source: §E.1 · p. 20

E.2. LLM Judge System Prompt

  • The judge evaluates experiment design, execution setup, implementation alignment, and conclusion correctness against the question and ground truth, returning an overall verdict and criterion-level assessments.Its requested format includes pass/fail judgments and explanations for failures.Source: §E.2 · p. 21

Related papers

Loading 2502.16069v2…