PaperlayerView exact revision on arXiv

Source-linked AI summary

DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, Peter Clark

arXiv:2406.06769v2cs.AIcs.CL

TL;DR

Real-world experiments make end-to-end scientific discovery costly to develop and evaluate. DISCOVERYWORLD provides a simulated benchmark of varied discovery tasks and automatic evaluation, and baseline agents struggle to complete most tasks or discover critical explanatory knowledge.

  • Problem

    Real-world experiments can be prohibitively expensive or infeasible, complicating development and evaluation of agents’ end-to-end scientific reasoning.

    Source: §1 · pp. 1–2

  • Method

    DISCOVERYWORLD is a simulated environment with diverse tasks requiring hypothesis formation, experimentation, analysis, and action on conclusions, evaluated through three automatic metrics.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Strong baseline agents struggle to complete most DISCOVERYWORLD tasks or discover critical explanatory knowledge.

    Source: §6 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    The benchmark is intended to support development of general AI discovery agents and their evaluation across end-to-end discovery tasks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Automated scientific discovery promises to accelerate progress across scientific domains. However, developing and evaluating an AI agent's capacity for end-to-end scientific reasoning is challenging as running real-world experiments is often prohibitively expensive or infeasible. In this work we introduce DISCOVERYWORLD, the first virtual environment for developing and benchmarking an agent's ability to perform complete cycles of novel scientific discovery. DISCOVERYWORLD contains a variety of different challenges, covering topics as diverse as radioisotope dating, rocket science, and proteomics, to encourage development of general discovery skills rather than task-specific solutions. DISCOVERYWORLD itself is an inexpensive, simulated, text-based environment (with optional 2D visual overlay). It includes 120 different challenge tasks, spanning eight topics each with three levels of difficulty and several parametric variations. Each task requires an agent to form hypotheses, design and run experiments, analyze results, and act on conclusions. DISCOVERYWORLD further provides three automatic metrics for evaluating performance, based on (a) task completion, (b) task-relevant actions taken, and (c) the discovered explanatory knowledge. We find that strong baseline agents, that perform well in prior published environments, struggle on most DISCOVERYWORLD tasks, suggesting that DISCOVERYWORLD captures some of the novel challenges of discovery, and thus that DISCOVERYWORLD may help accelerate near-term development and assessment of scientific discovery competency in agents. Code available at: www.github.com/allenai/discoveryworld

1 Introduction

Real-world discovery systems can be expensive and task-specific, while many successful systems bypass the full scientific process. DISCOVERYWORLD is introduced as a virtual benchmark for end-to-end discovery, spanning diverse tasks and providing automatic evaluation and baseline results.

1 Introduction

  • Real-world discovery systems can be expensive, and focusing on real environments may favor task-specific details over general discovery skills.Existing specialized systems often search predefined hypothesis spaces and goals rather than conducting the full process.Source: §1 · pp. 1–2
  • DISCOVERYWORLD is a simulated environment in which agents form hypotheses, run experiments, analyze observations, and draw conclusions across eight diverse scientific topics.Its simplified but realistic tasks are intended to encourage general rather than task-specific discovery skills.Source: §1 · p. 2
  • Automatic assessment of an agent’s progress on discovery tasks is itself difficult, motivating a three-part evaluation strategy.The introduction points to task completion, task-relevant actions, and explanatory knowledge as evaluation dimensions.Source: §1 · pp. 2–3
  • The introduction adopts a pragmatic meaning of scientific discovery: the end-to-end research process scientists use to address problems or advance their field.The paper notes that a precise definition is elusive.Source: §1 · p. 2
  • The paper contributes a 120-task benchmark, an automatic evaluation framework, and baseline results showing that contemporary agents struggle with its discovery challenges.The tasks span three difficulty levels and parametric variations.Source: §1 · p. 3

2 Related Work

Prior discovery systems include expensive, task-specific real-world approaches and virtual environments that cover narrower activities. DISCOVERYWORLD is positioned as a virtual environment for the full end-to-end discovery pipeline across diverse tasks.

2 Related Work

  • Real-world discovery systems have succeeded in genetics, chemistry, and proteomics, but are described as expensive and task-specific.DISCOVERYWORLD aims to provide broad coverage for developing and evaluating general scientific research skills.Source: §2 · p. 3
  • Other virtual environments cover activities such as navigation, object manipulation, entertainment-oriented exploration, or hypothesis search.The paper distinguishes these from its aim of covering the full end-to-end discovery pipeline.Source: §2 · p. 3
  • ScienceWorld focuses on elementary science quests and commonsense object manipulation, whereas DISCOVERYWORLD includes ideation and systematic search.The paper reports that DISCOVERYWORLD tasks can challenge scientists with PhDs in natural sciences.Source: §2 · p. 3
  • Table 1 compares DISCOVERYWORLD with existing virtual environments, with action counts treating each move as one action.The comparison is assembled from source materials and is presented as the authors’ best estimate.Source: Table 1 · p. 4
  • MLAgentBench uses software experiments to improve machine-learning algorithms, while DISCOVERYWORLD targets diverse tasks in a simulated physical environment.The paper characterizes DISCOVERYWORLD as covering the full end-to-end discovery pipeline.Source: §2 · p. 4

3 DISCOVERYWORLD Simulator and Environments

DISCOVERYWORLD combines a turn-based simulation with parametrically generated discovery tasks, unit tests, and automatic measures of task progress and explanatory knowledge.

3.1 Simulator

  • A custom Python/Pygame engine provides an API and graphical interface, with each turn presenting an observation and requiring one action.Objects have measurable material properties, some observable with in-world instruments.Source: §3.1 · pp. 4–5
  • The world uses a 32 × 32 tile grid, text or visual observations, and 14 actions, including teleportation aids for agent navigation.Text observations include nearby objects, inventory, interactable objects, location, and task status; visual observations show a larger area.Source: §3.1 · p. 5

3.2 Discovery Task Themes, Difficulty Levels, and Parametric Variations

  • Eight discovery themes each have three difficulty levels, with five deterministic seeds per combination yielding 120 benchmark task instances.Parametric variations can change task solutions, object properties, data, and starting locations.Source: §3.2 · p. 5

3.3 Unit Tests

  • Ten unit-test themes cover routine navigation and object tasks as well as discovery-related competencies; Reactor Lab scorecards include procedural steps and explanatory checks.The example checks whether crystal frequency depends linearly on densitometer readings, with frequency = (96 * densitometer reading) + 102.Source: §3.3 · pp. 5–6, Table 3 · p. 6

3.4 Evaluation Metrics

  • Evaluation combines binary task completion, a task-relevant action report card, and accuracy of discovered explanatory knowledge against a gold reference.Explanatory knowledge can be graded manually or by a language model; the paper’s evaluation uses GPT-4o.Source: §3.4 · p. 6

4 Experiments, Baseline Agents, and Human Baselines

The experiments evaluate zero-shot scientific-discovery performance with three baseline agents and compare task performance against practicing human scientists. The setup includes task-completion tables for discovery tasks and unit tests, plus a human evaluation.

4.1 Experimental setup

  • The evaluation reports baseline model performance on task completion, task process, and explanatory knowledge discovery across all 24 DISCOVERYWORLD tasks and all 10 unit-test tasks.Values average performance across 5 parametric seeds. Easy discovery tasks and unit-test tasks run to a maximum of 100 steps; Normal and Challenge discovery tasks run to 1000 steps.Source: Table 4 · p. 7, Table 5 · p. 7

4.2 Baseline Agent Models

  • ReAct produces a thought and action at each step from recent trajectory and game-state information, trimming older steps when needed to fit its prompt.In practice, the trajectory included up to the last 40 steps.Source: §4.2 · p. 8
  • Plan+Exec iteratively generates one plan step from prior steps and their success, then executes that step with the ReAct agent.This approach is intended to reduce distractors by keeping each ReAct trajectory smaller.Source: §4.2 · p. 8
  • Hypothesizer maintains explicit working memory for hypotheses and measurements, while its actions are guided by a brief plan and a running hypothesis.Its stored knowledge also supports more direct evaluation of discovered knowledge.Source: §4.2 · p. 8

4.3 Human Evaluation

  • Eleven practicing human scientists were recruited to complete DISCOVERYWORLD tasks for comparison with agent performance.Participants had a natural-science MSc or PhD, statistical-methods and spreadsheet fluency, and comfort with 2D top-down games.Source: §4.3 · p. 8

5 Discussion

Human scientists solved many DISCOVERYWORLD tasks, while baseline agents struggled with most tasks and explanatory knowledge. The results distinguish task completion from producing the explanatory discoveries required by the benchmark.

Human Performance

  • 66% was the human average task completion rate across tasks, while average knowledge performance was 55%.Eleven of 16 tasks completed by humans had completion rates above 60%, but several Challenge tasks were solved by only one participant.Source: §5 · p. 9

Agent Performance

  • 38% of easy tasks and 18% of challenge tasks were completed by ReAct, while Hypothesizer discovered 34% and 8% of gold knowledge in those respective difficulties.Unit-test completion rates in the 60%+ range suggest agents can perform components such as measuring objects, while they lack end-to-end discovery capacity in most settings.Source: §5 · p. 9

6 Conclusion

The conclusion presents DISCOVERYWORLD as a virtual benchmark for end-to-end scientific discovery, with 120 tasks requiring a full cycle of research actions. Human experts found challenge tasks difficult but solvable, whereas strong agent baselines struggled to complete most tasks or discover critical explanatory knowledge.

6 Conclusion

  • DISCOVERYWORLD provides 120 tasks requiring agents to form hypotheses, design and run experiments, analyze results, and act on conclusions.The paper presents it as the first virtual environment for developing and benchmarking end-to-end scientific-discovery ability.Source: §6 · p. 9
  • Expert human scientists found the challenge tasks difficult but solvable, while strong agent baselines struggled to complete most tasks or discover critical explanatory knowledge.Source: §6 · p. 9

Checklist

The checklist points readers to supplementary material for limitations, broader impacts, additional environment and baseline details, and human-participant information.

Checklist

  • The checklist marks limitations and broader impacts as discussed in Appendix B.Source: Checklist · p. 13
  • The checklist says the code repository provides the relevant image-asset license and attribution, and that anonymized human-participant trajectories and notes are released.Source: Checklist · p. 13

A Overview

  • The supplement overview assigns further environment details to Appendix C, baseline details to D, explanatory-knowledge evaluation to E, and human participants to F.Source: A Overview · p. 14

B Limitations and Broader Impacts

The paper identifies low simulation fidelity and high evaluation costs as constraints, and discusses both potential societal benefits and risks. It also notes that the environment and its action space are available for further work.

B Limitations and Broader Impacts

  • Low fidelity and an abstracted action space mean success in DISCOVERYWORLD may not transfer to real-world discovery, despite shared scientific-method steps.The authors point to the larger real-world search and action space as a reason agents may not perform equally well outside the simulation.Source: B Limitations and Broader Impacts · p. 15
  • Evaluating all 120 tasks with the tested agents costs approximately USD$3k-$10k, largely because runs can exceed 1,000 steps and use per-token-priced API models.The authors identify long runtimes, task count, and costly API-based models as cost drivers.Source: B Limitations and Broader Impacts · p. 15
  • Automated discovery could accelerate progress in medicine, chemistry, and technology, but harmful-intent users could also use it to cause harm.The authors frame both outcomes as potential impacts, dependent on users' intentions.Source: B Limitations and Broader Impacts · p. 15

C Additional Details on DISCOVERYWORLD

C.1 Action Space

  • The action space has 14 main actions, with two additional handicap actions intended to assist agents with navigation.Actions may take zero, one, or two arguments.Source: §C.1 · p. 15, Table 7 · p. 15

C.2 Experimental Settings

The appendix recommends training and evaluation splits, describes the benchmark's discovery themes and unit tests, and details baseline-agent procedures. Together, these passages specify how to run and interpret the environment's varied tasks.

C.2 Experimental Settings

  • Zero-shot agents can be evaluated independently on all 120 tasks; recommended learning splits separate training, development, and test seeds or themes, while curriculum learning follows increasing difficulty.The provided baselines use zero-shot evaluation; unit tests are available for training in the recommended learning settings.Source: §C.2 · pp. 15–16

C.3 Discovery Themes

  • Proteomics tests outlier detection, Chemistry tests solution-space search with a hill-climbing signal, and Archaeology tests radioisotope dating and validation.Proteomics difficulty increases clustering from two dimensions to three; Archaeology's challenge setting requires identifying a useful isotope and validating dating against known artifacts.Source: §C.3 · pp. 16–17
  • Reactor Lab tests regression to infer crystal-frequency relationships, Plant Nutrients tests rule inference from examples, and Space Sick tests open-ended investigation of illness causes.Challenge variants include quadratic relationships, logical nutrient rules, and discovering novel detection instrumentation, respectively.Source: §C.3 · p. 17
  • Rocket Science requires measuring planetary and propellant properties for orbital calculations, while Translation requires grounding unknown-language words in environmental clues.Translation challenge instructions combine a verb, amount, color, and item; easier settings simplify either the language task or the rocket calculation.Source: §C.3 · pp. 17–18

C.4.1 Unit Test Enviroments

C.5 Object Properties

  • The object-property table lists properties common to the base OBJECT class, while specialized objects may add properties such as a quantum crystal's resonance frequency.Source: §C.5 · p. 18, Table 9 · p. 20

D.1 ReAct

  • ReAct produces a thought and action at each step from an environment prompt and prior trajectory; long histories are trimmed to fit the prompt limit.The appendix says GPT-4o can reach its 8K-token limit in 40 steps and describes limiting trajectory history to 10K characters.Source: §D.1 · p. 21, D Additional Baseline Model Details · p. 21

D.2 Plan-and-Execute

  • Plan-and-Execute iteratively generates one plan step, runs it with ReAct, then uses the result to generate the next step.The prompt includes prior plan steps and results, and the ReAct agent may return a task-failed message when a step is infeasible.Source: §D.2 · pp. 21–22

D.4 Cost analysis

The cost analysis describes approximate baseline-model costs and the evaluation's upper-bound workload, while noting that costs vary with task and model-memory demands.

D.4 Cost analysis

  • Approximate model-cost averages vary by task, environment complexity, and stored memory; the estimates assume GPT-4o pricing of $5/M input tokens and $15/M output tokens.The cited table caption specifies those token rates as the cost assumption.Source: §D.4 · p. 26, Table 10 · p. 26
  • 84,000 steps is the upper-bound workload for a full benchmark run, assuming no early task completions.The estimate covers 40 easy tasks at 100 steps each and 80 normal or challenge tasks at 1,000 steps each.Source: §D.4 · p. 26

E Automatic Evaluation of Explanatory Discovery Knowledge

The scorer checks whether an agent’s knowledge answers a task-specific critical question about the required discovery. It returns a binary assessment with an explanation, illustrated by positive and negative examples.

E Automatic Evaluation of Explanatory Discovery Knowledge

  • The evaluator receives the task description, the agent’s knowledge, and one critical question, then judges whether the knowledge reflects discovery of the queried information.Its response includes the repeated question, a 0-or-1 evaluation, and a brief explanation referencing the agent’s knowledge.Source: E Automatic Evaluation of Explanatory… · p. 26
  • A positive example credits knowledge identifying the vortisquid’s protein levels as outliers, including Protein A: 0.38 and Protein B: 0.46.The example connects those measurements to the critical question about differences from other animals.Source: E Automatic Evaluation of Explanatory… · p. 27
  • Negative examples distinguish recording word-use or related interactions from explicitly stating the queried meanings of “Womple” and “florpt.”The scorer gives 0 when the agent’s knowledge does not directly state the requested translation or identification.Source: E Automatic Evaluation of Explanatory… · p. 27

F Human Scientist Participants

The human comparison recruited 11 practicing scientists to complete DISCOVERYWORLD tasks under specified time, interface, and evaluation conditions. Participants had relevant scientific and statistical experience, and some task data were lost because of save failures.

F Human Scientist Participants

  • Eleven practicing scientists with a natural-science MSc or PhD were recruited; screening required statistical-methods and common-software fluency, plus experience with 2D top-down games.Participants completed a DISCOVERYWORLD tutorial screening task, and their expertise spanned fields including biology, chemistry, geology, and physics.Source: F Human Scientist Participants · p. 28
  • Participants had up to one hour per task and used the 2D graphical interface; not all chose to complete every task.The interface example is for the “It’s (not) Rocket Science!” theme.Source: F Human Scientist Participants · p. 28, Figure 4 · p. 29
  • Human participants completed only normal and challenge settings, used the same task seed, and had retries excluded from zero-shot performance evaluation.They could retry unsuccessful tasks, but the retries were not counted in the evaluation.Source: F Human Scientist Participants · pp. 28–29
  • Task completion and procedural knowledge were evaluated automatically, while discovery knowledge was assessed manually from participant notes and hypotheses using a 0, 0.5, or 1 rubric.The rubric awarded 1 for all scorecard discovery knowledge, 0.5 for some, and 0 for missing critical knowledge.Source: F Human Scientist Participants · p. 28
  • Some Windows autosave failures caused a small amount of participant data to be lost and excluded from analysis.The reported save issues were infrequent and unresolved.Source: F Human Scientist Participants · p. 29
Loading 2406.06769v2…