ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows

1 INTRODUCTION

• Sophisticated scientific software and demanding workflows increase researchers’ learning and operational burdens, motivating interest in autonomous agents as AI co-scientists.The introduction gives protein-analysis work as an example of tasks that can take humans weeks to master and hours of observation.Source: §1 · p. 1
• Computer-using agents can automate scientific tasks through step-by-step interaction with operating systems via command-line commands or graphical mouse and keyboard actions.The paper illustrates GUI use with an agent launching ChimeraX and entering an amino-acid sequence into its AlphaFold widget.Source: §1 · p. 2
• ScienceBoard provides a realistic research-assistant environment integrating scientific software and standardized evaluation, with agents operating through either CLI or GUI.The environment is intended to help agents assist human scientists with daily tasks.Source: §1 · p. 2
• ScienceBoard’s benchmark contains 169 challenging tasks across six scientific domains, designed by disciplinary annotators to simulate scientists’ daily routines.Tasks require CLI or GUI interaction and capabilities including visual and textual reasoning, coding, mathematics, spatial understanding, and domain knowledge; evaluation functions check VM internal states.Source: §1 · p. 2
• Across observation settings, agents’ average success rates range from 0% to 15%, peaking at 20% in the most favorable subcategories.The authors attribute the shortfall largely to limited action capability and domain knowledge.Source: §1 · p. 2

Computer-Using Agents

• Computer-using agents use CLI scripts or APIs and GUI mouse-and-keyboard actions to automate computer tasks, including desktop, mobile, and engineering workflows [OpenAI, 2025; Wang et al., 2024b; Cheng et al., 2024; Cao et al., 2024].These approaches are presented as promising paths toward digital automation.Source: §2 · p. 3

AI for Scientific Discovery

• LLM/VLM research applications span idea generation, data analysis, programming, writing, review, and domain-specific tasks, while ScienceBoard targets end-to-end scientific exploration [Luo et al., 2025; Si et al., 2024].The related-work section describes this work as the first to support agents in executing end-to-end scientific exploration workflows.Source: §2 · p. 3

3 SCIENCEBOARD ENVIRONMENT

• ScienceBoard combines real-world scientific software in an Ubuntu VM with GUI and CLI interfaces, creating dynamic, visually intensive settings for agent interaction.The environment is designed to support autonomous exploration beyond static question answering, coding, or single-step tasks.Source: §3 · p. 3, §3.2 · p. 4

3.1 PRELIMINARIES AND TASK DEFINITION

• Agent interaction is modeled as a POMDP: a policy uses the goal and recent observation-action memory to choose actions, while environment dynamics determine the trajectory.Observations may be textual, visual, or combined; text representations use accessibility trees, and visual observations use high-resolution screenshots.Source: §3.1 · pp. 3–4
• The unified action space supports GUI inputs, system and application CLI commands, scripting, answer actions, and API calls.The policy receives the current observation and selects the next action; LLMs are used for text-only observations and VLMs otherwise.Source: §3.1 · p. 4

3.2 SCIENTIFIC DISCOVERY EVALUATION FRAMEWORK

• Scientific applications are adapted to expose internal runtime states, and task environments can be initialized with specific configurations for contextualized evaluation.Agents act in an observation-action loop until signaling DONE or FAIL, or reaching the attempt limit; evaluation checks workflow I/O and final VM state using task-specific templates.Source: §3.2 · p. 5
• Evaluation cases include exact matching, range-based assessment, and numerical tasks with tolerance.The authors tailor evaluation methods to the correctness criteria of each task.Source: Table 1 · p. 5, §3.2 · p. 5

4.1 DOMAIN AND TASK COVERAGE

• Tasks span biochemistry, algebra, theorem proving, GIS, astronomy, and scientific documentation, covering scientific reasoning, software use, and cross-application reporting.Documentation tasks can combine with other domains to turn completed upstream work into experimental reports.Source: §4.1 · p. 6
• The six domains assess visual and textual reasoning, mathematics, coding, tool use, spatial understanding, and domain knowledge.The selected software also meets practical criteria including open-source availability, a11ytree compatibility, and no authentication requirement.Source: §4.1 · p. 6

4.2 TASK ANNOTATION PIPELINE

• Annotators learn software from tutorials, draft tasks around its functions, then standardize and cross-check tasks to remove low-diversity, non-executable, or non-unique examples.Tasks include configurations, simulations, QA, and domain-specific work.Source: §4.2 · p. 7
• Each task receives initialization and evaluation functions; separate annotators execute tasks on separate virtual machines to cross-validate evaluator correctness.Evaluators inspect task state against criteria such as input/output matching and predefined ranges.Source: §4.2 · p. 7

4.3 TASK STATISTICS

• The benchmark contains 169 tasks across six domains, with three difficulty levels and a balance between representativeness and manageable evaluation cost.Its task types cover diverse operation flows and domain-specific knowledge.Source: §4.3 · p. 7, Table 2 · p. 7
• SCIENCEBOARD improves task-design and content diversity over prior work, with task types distributed across the benchmark.The benchmark’s distribution is shown in Figure 4.Source: §4.3 · p. 7

5.2 RESULTS

• 20% average success for GPT-5 and Gemini-2.5-Pro contrasts with open-source agents’ average below 12%, with some categories near 0%.The reported agent–human performance gap underscores the remaining limitations.Source: §5.2 · p. 8, Table 3 · p. 9
• Agents perform moderately well in Algebra and Biochemistry but worse in GIS and Astronomy, where GUI use and dense visual elements pose challenges.Fine-grained GUI grounding and limited 3D spatial reasoning are identified as contributing difficulties.Source: §5.2 · p. 8
• Screenshots plus a11ytree yield the best overall performance, while Set-of-Marks can hinder reasoning in visually dense scientific software.The authors attribute Set-of-Marks’ negative effects to noise from marked regions amid dense elements.Source: §5.2 · p. 8
• More inference-time compute brings bounded gains, including in Biochemistry and GIS, but does not resolve the overall performance deficit.The paper identifies accurate perception of dense domain-specific interfaces and conversion of plans into executable actions as key bottlenecks.Source: §5.2 · pp. 8–9

5.1 EXPERIMENTAL SETTINGS

• The evaluation compares proprietary, open-source, and GUI-action-model backbones across scientific domains and four observation settings.Settings include screenshots, a11ytree, their combination, and Set-of-Marks.Source: §5.1 · p. 8, Table 3 · p. 9

5.2 RESULTS

• Scientific agents need domain knowledge and software-specific expertise in addition to planning and action, and must flexibly combine GUI and CLI execution.CLI can provide a more concise route for complex operations when the agent knows how to use it.Source: §5.2 · pp. 9–10

GUI vs. Hybrid

• Disabling ChimeraX’s CLI causes clear performance drops for GPT-4o and InternVL3, while Qwen2.5-VL remains largely unaffected.This comparison tests GUI-only execution against hybrid GUI-plus-CLI access.Source: §6 · p. 10, Figure 5 · p. 10
• The authors argue future agents need stronger GUI capabilities and adaptability across hybrid and vision-only settings.The recommendation follows the differing responses to disabling CLI access.Source: §6 · p. 10

Disentangled Planning and Action

• Modular planner-and-grounder approaches yield significant improvements and appear promising for complex, visually demanding scientific workflows.GPT-4o serves as planner while VLMs or GUI action models provide grounding.Source: §6 · p. 10, Table 4 · p. 10

7 CONCLUSION

• SCIENCEBOARD combines a rigorously validated environment with a challenging benchmark of diverse scientific tasks curated by human experts.The authors position autonomous agents for scientific discovery as a distant goal and describe their analysis as offering actionable development insights.Source: §7 · p. 10

• Live operating systems could be disrupted by computer-using agents, but the authors do not consider this a concern for experiments conducted in isolated virtual environments.Source: ETHICS STATEMENT · p. 11

LIMITATIONS AND BROADER IMPACTS

• Binary success flags may miss process correctness or partial task completion, while implementing partial credit for open-ended OS tasks is challenging.The authors suggest VLM judges as a possible source of partial credit and richer feedback.Source: LIMITATIONS AND BROADER IMPACTS · p. 17

A DISCUSSION AND FUTURE DIRECTIONS

• Insufficient domain knowledge may limit agents in scientific exploration, motivating domain-specific learning and retrieval while harmonizing expertise with general agentic capability.Source: A DISCUSSION AND FUTURE DIRECTIONS · p. 18
• A basic modular division between planning and action can improve performance in complex scientific software workflows, though coordination in more sophisticated multi-agent systems remains challenging.Source: A DISCUSSION AND FUTURE DIRECTIONS · p. 18
• The authors propose extending digital agents to physical laboratories, where they could operate instruments and samples and monitor experiments in real time.Source: A DISCUSSION AND FUTURE DIRECTIONS · p. 18

B.1 ENVIRONMENT SETUP

• The experiments use Ubuntu 22.04.1 LTS with kernel 6.8.0-57-generic on x64 personal computers.Source: §B.1 · p. 18

B.2 EVALUATION CRITERIA

• Evaluation considers final VM state alongside I/O and intermediate-step correctness, using exact matches, line-set equality, answer matching, and predicate satisfaction.Source: §B.2 · pp. 18–19
• Additional criteria include required-value presence or absence, range checks with tolerances, correct failure reporting, and domain-specific success markers such as Lean proof compilation.Source: §B.2 · p. 19

B.3 SELECTION AND MODIFICATION OF SCIENTIFIC SOFTWARE

• Software selection prioritizes accessibility, GUI compatibility, and domain representativeness, with tools spanning GIS, scientific writing, theorem proving, molecular visualization, mathematics, and astronomy.Source: §B.3 · pp. 19–20

B.4 DETAILS OF ACTION SPACE

• The action space combines GUI operations with system- and application-level CLIs, augmented with actions tailored to scientific workflows.Source: §B.4 · p. 20, Table 5 · p. 20

B.5 DETAILS OF OBSERVATION SPACE

• Agents can receive screenshots, accessibility-tree text, a combined screenshot and tree, or Set-of-Marks visual prompts; the tree represents UI content through OS accessibility APIs.Source: §B.5 · pp. 20–21

C ACCESSING SCIENCEBOARD ENVIRONMENT

• SCIENCEBOARD can be deployed natively on Ubuntu, run through a Docker image, or hosted on cloud platforms such as AWS.Source: C ACCESSING SCIENCEBOARD ENVIRONMENT · p. 21

D.1 TASK ANNOTATION

• Authors used tutorials and handbooks to guide annotators with disciplinary backgrounds in exploring relevant domains and software, and completed app data collection and task creation themselves.Source: §D.1 · p. 21

D.2 TASK DIVERSITY

• Across three runs, Algebra task performance remains stable, while Biochemistry success rates fluctuate slightly, with network issues or transient system lag offered as possible explanations.Source: §D.2 · p. 21

D.3 COMPARISON WITH EXISTING BENCHMARKS

• SCIENCEBOARD is presented as the first realistic environment for scientific-task evaluation, incorporating structured code input, visual information, and GUI automation.Source: §D.3 · p. 21, Table 7 · p. 23

D.4 MORE EVALUATION SCRIPT EXAMPLES

• The appendix provides additional evaluation pipelines beyond the cases listed in the main evaluation criteria.Source: §D.4 · p. 23

D.5 HUMAN PERFORMANCE

• College-level participants familiarized themselves with relevant foundational knowledge and operational manuals before attempting benchmark tasks.Source: §D.5 · p. 23

D.6 STABILITY ANALYSIS

• Across three independent runs, Algebra performance remains stable, while Biochemistry success rates fluctuate slightly, possibly due to network issues or transient system lag.Source: §D.6 · p. 23, Figure 7 · p. 23

D.7 EVALUATION COST

• A single run over all SCIENCEBOARD tasks costs $64 with GPT-4o, $86 with Claude-3.7-Sonnet, and $45 with Gemini-2.0-Flash.Source: §D.7 · p. 24

E.1 BACKBONE MODELS

• The evaluated backbones include open-source general models, GUI action models, and proprietary models accessed by API.Source: §E.1 · pp. 24–25

E.2 EVALUATION SETTINGS - MAIN EXPERIMENTS

• Agents follow domain-specific meta-prompts and ReAct-style actions; evaluations use temperature 0.5, top_p 0.9, and max_tokens 1500.The meta-prompts describe the software and executable special actions for each domain.Source: §E.2 · p. 26

E.3 EVALUATION SETTINGS - ANALYSIS

• The analysis normalizes GUI coordinates to [0,1] and uses a planner followed by a grounding model unless the planner emits a directly executable action.Native coordinate outputs from different GUI models are normalized before execution.Source: §E.3 · p. 26

F.1 INTERFACES

• Hybrid GUI + CLI matched or exceeded GUI-only performance across scenarios, while GPT-4o struggled with screenshot-only interaction.The comparison is reported for the other observation settings.Source: §F.1 · p. 27, Figure 8 · p. 27

F.2 INTERACTIVE ENVIRONMENTS

• Interactive ATP adds live VSCode navigation and PROOFSTATE selection, making formal-methods tasks substantially more cognitively complex than textual proof-tree heuristics.The interactive setting also requires locating relevant code segments in the interface.Source: §F.2 · p. 27, Figure 9 · p. 27

F.3 DIFFICULTY ANALYSIS

• No evaluated agent completed any hard task; successes concentrated in Easy tasks and a few Medium tasks.Figure 10 presents the comparative task-difficulty solve rates.Source: §F.3 · p. 28, Figure 10 · p. 28