PaperlayerView exact revision on arXiv

Source-linked AI summary

MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows

Mei Wu, Rui Xie, Runyu Zhang, Yuqiang Li, Tianfan Fu, Bo Chen, Kai Yu, Xin Chen, Lu Chen

arXiv:2609.37053v1cs.AI

TL;DR

Professional scientific software remains an underexplored setting for multimodal agents, despite their progress on general software benchmarks. MatToolBench evaluates agents across materials-science tools and workflows; results show low success rates on GUI and code tasks. The benchmark offers a diagnostic testbed for these workflows.

  • Problem

    How well multimodal agents operate professional scientific software remains largely unexplored, amid sparse domain-specific web data and specialized interfaces and workflow conventions.

    Source: Introduction · p. 1

  • Method

    MatToolBench evaluates agents on 204 materials-science tasks across 10 tools and GUI, OriginPro, and database-query workflows, using expert-defined sub-criteria and modality-specific evaluation.

    Source: Introduction · p. 2

  • Results

    The best models reach only 25% SR on GUI tasks and 45% SR on code tasks, providing diagnostic evidence of a gap between general benchmarks and scientific software.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    MatToolBench provides a practical testbed for diagnosing scientific-agent performance, with workflow support shown to matter especially when interfaces and APIs are sparsely represented in pretraining.

    Source: Conclusion · p. 8

    The benchmark covers a curated set of ten tools and single-session interactions of at most 50 steps, excluding longer multisession or multi-agent pipelines.

    Source: Appendix R · p. 25

Abstract

from arXiv · show

Multimodal GUI agents have achieved impressive results on general software benchmarks, yet their ability to operate professional scientific software remains largely unexplored. In materials science, sparse domain-specific web data, specialized interfaces, and tacit workflow conventions create blind spots that general-purpose pretraining cannot readily bridge. We present MatToolBench, the first real-environment benchmark for evaluating multimodal GUI agents on professional materials science software, comprising 204 tasks across 10 tools in three modalities: GUI operation, OriginPro scripting, and code-based database queries, all executed inside a Windows 11 VM. Each task is decomposed into fine-grained sub-criteria by domain experts, enabling interpretable partial-credit scoring; the GUI component of our multi-level evaluation pipeline achieves an average F1 of 0.98. For OriginPro figure-generation tasks, we further conduct a human-LLM agreement study to validate the use of a multimodal judge for secondary aesthetic assessment. Our experiments show that strong performance on general benchmarks does not transfer to professional scientific workflows, and that this gap is not a visual-grounding problem alone: failures arise from domain-specific operational knowledge, sparse pretraining coverage of scientific software, weak cross-tool artifact handoff, and critical states exposed only visually. Even the best model reaches only 25% success rate on GUI tasks and 45% on code tasks. MatToolBench therefore serves as a challenging diagnostic benchmark and real-environment testbed for data-scarce, knowledge-intensive scientific workflows.

Introduction

MatToolBench addresses the open question of whether multimodal agents can handle professional materials-science workflows, which combine specialized interfaces, domain knowledge, and cross-tool work. Its benchmark evaluation finds low success rates and shows that performance depends on workflow guidance.

Introduction

  • Whether agents can perform complex real-world scientific workflows remains open because pretraining data for professional scientific software is scarce.Prior GUI-agent research is predominantly confined to everyday general-purpose software.Source: Introduction · p. 1
  • Materials workflows combine heterogeneous tools, cross-tool artifact handoffs, and information-dense interfaces with implicit feedback and deep menus.Some analyses require moving raw XRD data from JADE to OriginPro for plotting and annotation; these mixed workflows are diagnostic rather than the main task split.Source: Introduction · p. 1, Figure 2 · p. 2
  • MatToolBench provides 204 real-environment tasks across 10 tools and GUI, OriginPro scripting, and database-query modalities, with expert-defined criteria for partial-credit scoring.Tasks run in a Windows 11 VM with the target software pre-installed.Source: Introduction · p. 2
  • The GUI evaluator achieves average F1 of 0.98; OriginPro scoring combines deterministic export checks with an LLM aesthetic judge validated against human ratings.Code tasks are evaluated by matching reference outputs when deterministic answers exist.Source: Introduction · p. 2
  • The best model reaches only 25.0% GUI SR and 45.0% code SR; Origin SR ranges from 0.0% to 68.8% across models.DigitalMicrograph stays at or below 20.0% SR for every model, while OPTIMADE reaches up to 85.0%.Source: Introduction · p. 2
  • Removing hints cuts JADE SR from 25.0% to 5.0% and GPT-5.4 MP performance from 15.0% to 0.0%; removing full Origin workflow support drops Claude-sonnet-4.6's score from 17/32 to 5.2/32.The reported ablations point to the importance of domain-specific workflow guidance.Source: Introduction · p. 2

Related Work

Prior GUI-agent benchmarks span web, mobile, and desktop settings, while scientific-agent benchmarks increasingly address executable tools and multi-step workflows. The supplied review identifies materials-science workflows and visually exposed software state as remaining gaps.

GUI-based Agents

  • GUI-agent benchmarks have expanded from web and mobile interaction to desktop environments, but do not target materials-science workflows or their domain-specific evaluation criteria.Broad professional-software suites and work on software-specific knowledge are also discussed, without targeting materials-science tool chains.Source: GUI-based Agents · p. 2

Science Benchmarks

  • Scientific-agent benchmarks extend from textual reasoning to executable tools and multi-step workflows, while MatToolBench targets software state often exposed only visually.The reviewed benchmarks mostly evaluate structured states, scripts, or logs.Source: Science Benchmarks · p. 2

MatToolBench Environment

MatToolBench runs GUI, code, and OriginPro workflows through a unified interface in a Windows 11 virtual machine. Tasks use reproducible initialization, modality-specific observations and actions, and post-episode scoring.

Task Definition

  • Tasks are modeled as partially observable episodes conditioned on the instruction and interaction history, ending on DONE, FAIL, or the default 50-step limit.Initialization brings each task to a deterministic starting state, and independent sub-criteria enable partial credit.Source: Task Definition · pp. 2–3

MatToolBench Environment

  • A central runner dispatches GUI, code, and Origin agents through a unified evaluation interface in a Windows 11 VM.The VM sits inside a Docker container and communicates through a Flask control server.Source: MatToolBench Environment · p. 2, Figure 3 · p. 3

Observation Space

  • GUI and Origin observations include a full-desktop screenshot; GUI observations also include the instruction, foreground-window metadata, and clipboard, while code tasks receive text execution feedback only.Screenshots are used because accessibility metadata may be incomplete or unavailable in legacy scientific software.Source: Observation Space · p. 3

Action Space

  • GUI agents act through mouse, keyboard, clipboard, program-control, and window-management tools; code agents generate scripts, while Origin agents combine scripting with GUI navigation.Origin scripts are injected into Code Builder via clipboard paste and F5.Source: Action Space · pp. 3–4

Execution Environment

  • An outer Linux Docker container hosts agent logic and evaluation services, while an inner Windows 11 VM holds target applications and isolated Python environments.A Flask server in the VM provides the sole control interface.Source: Execution Environment · p. 4

The MatToolBench Benchmark

The benchmark covers ten materials-science tools with tasks stratified by difficulty and decomposed into verifiable criteria. Its evaluation extracts task results and scores partial completion, success, and efficiency.

Task Statistics

  • MatToolBench covers 204 tasks across 10 tools, with 454 single-tool scoring criteria and 30 mixed diagnostic criteria.Mixed tasks are reported separately as cross-tool diagnostics.Source: Task Statistics · p. 4, Table 1 · p. 4
  • Tasks are stratified into easy, medium, and hard levels; GUI difficulty uses criterion counts of 1–3, 4, and 5–7, respectively.Origin and code difficulty labels reflect workflow complexity; mixed tasks use the point-count proxy.Source: Task Statistics · p. 4

Evaluation Protocol

  • After each episode, a domain-specific getter extracts results and a metric function compares them with a reference; Score measures partial credit and SR requires all criteria.Steps, successful-episode Steps*, and false termination rate (FTR) capture efficiency and termination behavior; step counts have a 50-step maximum.Source: Evaluation Protocol · p. 4

Experiments

Experiments evaluate seven multimodal models under a controlled setup and find low success across professional materials-science workflows. Ablations show that workflow guidance matters most where tool conventions are less self-evident or require scripting.

Experimental Setup

  • Episodes used API-accessed models, a 50-step budget, and reset VM states with fixed resolution, inputs, and initialization.GUI and Origin models received raw screenshots through NaviAgent; code tasks had no visual input.Source: Experimental Setup · pp. 4–5

Main Results

  • 24–25% GUI SR and 41–45% code SR were the best model ranges; GPT-5.4 had the best overall average, followed by Claude-sonnet-4.6.The reported general desktop benchmark success rates exceed 70%, while the best MatToolBench GUI SR is 25.0%.Source: Main Results · p. 5, Table 2 · p. 5
  • DM remained uniformly difficult, with no model exceeding 20% SR; on Origin tasks, GPT-5.4 led at 68.8%, while Qwen3-VL-8B scored 0%.MS had the lowest average SR across models.Source: Main Results · pp. 5–6
  • Across GUI domains, Score averages ≈50% while SR is ≈20%, showing models complete a fraction of scoring sub-criteria but rarely satisfy all simultaneously, revealing meaningful progress despite falling short of full completion.GUI tasks have high FTR values (40–70%), indicating persistent premature termination across all models.Source: Main Results · pp. 5–6

Ablation Study

  • Removing hints reduced GUI SR, including Doubao-seed-1-8’s JADE result from 25.0% to 5.0% and DM from 20.0% to 0.0%; standardized OPTIMADE was less affected.The ablation excluded Avantage and VESTA because their workflows were considered self-evident; niche or unstable API conventions benefited more from hints and examples.Source: Ablation Study · pp. 6–7, Figure 4 · p. 6
  • 3-level Origin ablation reduced deterministic success from 9 to 3 points at baseline, a 67% reduction, while combined score fell from 17/32 to 5.2/32.Removing the template script mainly affected visual quality; removing workflow hints as well often led to GUI-only exploration.Source: Ablation Study · p. 7, Figure 5 · p. 7

Evaluator Reliability

The evaluation uses separate reliability checks for GUI, code, and Origin tasks. For Origin, deterministic export checks define success, while a validated LLM judge supplies a secondary aesthetic score.

GUI Evaluator

  • GUI evaluator reliability is assessed by applying each domain’s evaluators across 20 final states and reporting Score and SR agreement; DM uses a filtered audit.The cross-evaluation also measures matched detection and cross-task false positives.Source: GUI Evaluator · p. 7, Table 4 · p. 7

Code Evaluator

  • Code checks use reference-file comparison for Pymatgen and tolerance-based value matching for MP/OQMD, with parseability, validity, and task relevance also checked.OPTIMADE is assessed through constrained queries and saved valid parsed values rather than a single gold file.Source: Code Evaluator · p. 7

Origin Aesthetic Evaluator

  • Origin SR depends on deterministic export-path checks; Gemini-3.1-pro provides only secondary quality scores, and its correlation with Qwen3-VL-235B did not differ significantly.The comparison used 20 human-rated conditions and 43 generated figures; the figure evaluates visual correctness, aesthetic quality, and task completeness across four task types.Source: Origin Aesthetic Evaluator · p. 7, Figure 6 · p. 7

Conclusion

MatToolBench provides diagnostic evidence of a transfer gap to scientific software: the best models achieve only 25% success on GUI tasks and 45% on code tasks. Its task schema, distribution, taxonomy, and action interface support reproducible, fine-grained evaluation across scientific workflows.

Conclusion

  • The best models achieve only 25% success on GUI tasks and 45% on code tasks, while benchmark results diagnose a gap between general benchmarks and scientific software.The benchmark offers a practical testbed, and its ablations indicate that workflow support is crucial, especially for tools and APIs sparsely represented in pretraining.Source: Conclusion · pp. 7–8

Appendix A. Task Configuration Schema

  • The shared task schema separates agent instructions, VM initialization, runtime metadata, and sub-criterion evaluators to make execution reproducible and task assignment explicit.Mixed tasks add stages specifying domains, agent types, goals, and artifacts; the same schema serves GUI, code, Origin, and mixed tasks with domain-specific setup and evaluators.Source: Appendix A · p. 10

Appendix B. Additional Task Distribution

  • Task distribution is intentionally non-uniform: GUI-intensive tools receive finer categories because visual states, modal dialogs, and interaction conventions shape their difficulty.The figure presents domain proportions in its outer ring and task-family composition in inner segments.Source: Distribution and Taxonomy · p. 10, Appendix B · p. 10, Figure 7 · p. 10

Distribution and Taxonomy

  • The taxonomy covers spectroscopy, crystal-structure, and database workflows, letting partial-credit criteria assess intermediate actions as well as final scientific outputs.Tasks include operations such as spectroscopy processing and export, crystal display or editing, and database filtering, retrieval, and output validation.Source: Distribution and Taxonomy · p. 10

Appendix C. Complete Action Interface

  • The Computer object exposes a unified Python API for VM interaction through HTTP POST, with actions covering mouse, keyboard, clipboard, program control, and window management.The interface is summarized in Table 6.Source: Appendix C · p. 10, Table 6 · p. 11

Appendix D. Execution Environment and Reproducibility

The execution setup standardizes task initialization and runtime conditions across evaluations, while isolating parallel workers. It combines GUI/Origin screenshot agents with a text-only code agent and evaluates outcomes against task-specific criteria.

Execution Environment

  • Evaluations run in Docker on Azure ML Compute Instances, with an inner QEMU-based Windows 11 VM and fixed compute resources.The stated instance type is Standard_D8_v3, with 8 vCPUs and 28 GiB RAM.Source: Appendix D · p. 12

Episode Initialization

  • Each task initializes from its JSON configuration, resetting the VM to a task-specific state before the agent acts and allowing up to 50 steps.Post-hoc evaluation occurs after termination, so intermediate rewards do not alter the agent policy during a run.Source: Appendix D · p. 12

Parallelism and Isolation

  • Parallel workers use separate compute instances or copy-on-write VM overlays, preventing shared application state and generated output files.On-demand instances are deleted after jobs complete.Source: Appendix D · p. 12

Agent Wrappers

  • GUI and Origin agents use screenshots in a shared NaviAgent loop, while code tasks use a text-only agent that runs generated Python in the task environment.The GUI agent calls the Computer API; the Origin agent can also generate longer script fragments.Source: Appendix D · p. 12

Evaluation and Aggregation

  • A task counts as successful only when every required sub-criterion is satisfied; its score sums partial-credit sub-criteria normalized by available points.Origin success uses deterministic output-file checks, with the LLM aesthetic judge contributing only a secondary quality score.Source: Appendix D · p. 12

Reproducibility Controls

  • Reproducibility controls standardize the VM and Docker images, setup scripts, inputs, decoding settings, step budget, and post-action wait; released artifacts support redeployment.The package includes task JSON, input artifacts, gold/reference outputs, evaluator definitions, and Origin reference scripts.Source: Appendix D · p. 12

Appendix E. Complete Task List

Appendix E documents MatToolBench’s individually specified single-tool tasks across GUI, OriginPro, and code-based materials workflows, alongside representative cross-tool workflows and the controlled scientific files used to run them. It also describes a modular evaluation protocol for scoring final states and exposing interpretable failure points.

Appendix E. Complete Task List

  • The appendix lists 196 single-tool tasks; entries pair a tool/domain and difficulty tier (easy, medium, or hard) with a shortened instruction.The eight mixed cross-tool diagnostic tasks are reported separately in the main text.Source: Appendix E · pp. 12–13
  • GUI tasks cover scientific operations in Avantage, DigitalMicrograph, JADE, Materials Studio, VESTA, and Origin, from spectral analysis to structure editing and figure creation.Examples include peak fitting, FFT analysis, crystallographic refinement, atomistic modeling, and annotated plots.Source: Appendix E · p. 13
  • Code-based tasks exercise Materials Project, OQMD, and OPTIMADE queries alongside Pymatgen structure creation, parsing, conversion, and analysis.Listed operations include filtering materials by composition or properties and extracting energies, tensors, structures, and band gaps.Source: Appendix E · p. 14
  • Mixed tasks require stage-wise handoffs across software boundaries, combining database retrieval or Pymatgen processing with GUI editing or modeling.Examples include exporting a Materials Project structure for VESTA, passing OPTIMADE output to Pymatgen, and moving generated structures into Materials Studio or VESTA.Source: Appendix E · pp. 14–15, Figure 8 · p. 15
  • The task list includes Avantage XPS reporting and plotting, JADE XRD matching followed by Origin plotting, and LiCoO2 structure generation with separate VASP input directories.The listed tasks involve exporting an Avantage report, matching an XRD PDF card, and generating structures for three Li contents.Source: Appendix E · p. 15

Appendix F. Mixed Cross-Tool Diagnostic Tasks

  • The eight mixed tasks probe cross-tool handoffs: code-generated structures opened in GUI tools and GUI-processed spectra or tables transferred to Origin.The handoff depends on saving an artifact with the correct path, extension, and semantic content for the next tool.Source: Appendix F · p. 15

Appendix G. Bundled Dependency Files

  • The VM provides curated authentic experimental and simulation files in native formats, with controlled inputs that reduce setup overhead and evaluation variance.The files cover GUI- and code-oriented workflows and encode scientific conventions such as XPS scan names and chemically meaningful structure information.Source: Appendix G · pp. 15–16

Appendix H. Evaluation Protocol Details

  • Evaluation uses domain-specific getters and metrics, including binary GUI and Origin checks, tolerant key-value matching for MP and OQMD, and valid-output checks for OPTIMADE.GUI getters combine OCR, pixel analysis, SSIM, dialog recognition, and domain-specific predicates because software exposes state through different channels.Source: Appendix H · p. 16
  • The evaluation pipeline scores task criteria from captured final states, independent of action sequence, enabling partial credit and distinguishing operational failures from domain-reasoning failures.It can identify missing states such as an unopened file, unset parameter, unsaved plot, or unwritten numeric field.Source: Appendix H · p. 17

Appendix I. Evaluator Reliability Details

The reliability audit tests GUI getters against matching and non-matching task configurations and reports both sub-criterion and task-level F1. It also examines visual disagreements and validates the Origin aesthetic judge against human ratings.

GUI Getter Reliability

  • The GUI cross-evaluation tests 20 final states per domain against every task configuration in that domain to measure successful detection and cross-task false positives.Score F1 uses binary sub-criterion partial-credit labels; SR-F1 uses task-level binary success labels.Source: Appendix I · p. 18
  • Reliability filtering excludes microscopy cues that are valid getters but nondiscriminative across tasks; this changes only the audit, not benchmark evaluators or task scores.Excluded cues include common short OCR tokens, shared geometry checks, and file-import labels with matching filename stems but different extensions.Source: Appendix I · p. 18
  • Residual GUI disagreements mainly involve OCR confusion, transient dialogs or highlights, and rendering-sensitive pixel thresholds; modular getters trace disagreements to individual sub-criteria.The paper describes these edge cases as rare and says they do not change the conclusion that automated getters closely track expert annotations.Source: Appendix I · pp. 18–19

Origin Aesthetic-Judge Reliability

  • All four Origin judges parse all 43 figures; Qwen3-VL-235B has the highest correlations, but its ratings are more saturated and lenient than Gemini-3.1-pro's.Their correlations are not statistically distinguishable in the reported test (∆r = 0.043, t = 0.582, p = 0.569); Gemini-3.1-pro is used for secondary aesthetic scoring, not Origin SR.Source: Appendix I · p. 18, Table 14 · p. 19

Appendix J. Efficiency Results

Efficiency measures distinguish task effort from premature stopping: GUI tasks take more steps than code tasks, while false termination remains common on GUI tasks. Steps* and FTR complement raw step counts by separating successful-run effort from self-terminated failures.

Appendix J. Efficiency Results

  • 35–42 steps for GUI tasks versus 1–4 for code tasks; GUI tasks also show 40–70% FTR, indicating persistent premature termination.Steps is the mean across episodes; FTR is the fraction of agent-terminated episodes with SR = 0.Source: Appendix J · p. 19
  • Steps* averages step counts only on successful episodes, while FTR measures failure only among runs the agent terminated itself.Steps* is left undefined when no episode succeeds.Source: Appendix J · p. 19
  • Raw step counts can look efficient when an agent declares completion before the required scientific operation, so Steps* and FTR offer complementary views of efficiency and reliability.Source: Appendix J · p. 19

Appendix K. Origin Ablation Output Examples

These appendices distinguish figure quality from artifact generation, describe final-state evaluation and visual checks, and examine recurring GUI and domain-knowledge failures.

Appendix K. Origin Ablation Output Examples

  • Template scripts produce high-quality Origin figures (s ≥0.98), while GUI-only execution yields s ≤0.70 or fails (s = 0).The quality score combines correctness, aesthetics, and completeness; template scripts improve scaling, annotations, and formatting.Source: Appendix K · pp. 19–20, Figure 9 · p. 20

Appendix L. Evaluation Scoring Criteria Examples

  • The evaluation algorithm extracts evidence for each criterion from the final VM state, assigns weighted criterion scores, and records success only when every criterion passes.Source: Appendix L · p. 20

Avantage Task: Step-by-Step Evaluation

Detailed Avantage Case Study

  • The Avantage example scores seven milestones independently, granting partial credit while exposing which scientific state or saved artifact is missing.Criterion-level traces also help domain experts audit evaluator errors and adjust affected getter thresholds.Source: Detailed Avantage Case Study · p. 21

Appendix M. Visual Pixel Detection Methods

  • Visual evaluators use raw screenshot pixels for cues such as radio-button position, icon color, structural similarity, and drawing handles, then map detections to binary scores.These checks address GUI states for which text labels or accessibility metadata are absent or insufficient.Source: Appendix M · pp. 21–22, Figure 11 · p. 22

Appendix N. Agent System Prompts / Appendix O. Software and Tool Descriptions

  • The benchmark uses tailored prompts across ten tools and four interaction modalities, from GUI-only operation to embedded scripting, APIs, and file-based structure analysis.Source: Appendix N · p. 22, Appendix O · p. 23

GUI Grounding Errors

Premature Termination / Deadlock / Repetitive Loops / Domain Knowledge Gaps

Appendix Q. Contrast with General GUI Benchmarks

The benchmark comparison is diagnostic rather than a controlled leaderboard, and the paper identifies both scientific-workflow differences and scope boundaries.

Appendix Q. Contrast with General GUI Benchmarks

  • General GUI agents achieve higher success on broad desktop or Windows tasks than on MatToolBench, but differences in tasks, wrappers, observations, actions, and evaluators prevent controlled head-to-head claims.Source: Appendix Q · p. 25, Table 16 · p. 25
  • MatToolBench centers on preserving scientific state across specialized interfaces, domain file formats, instrument-like visual feedback, and software-specific conventions.The cited MMBench-GUI results also show below-30% average success on long-horizon single-app tasks and below-10% for best cross-app collaboration.Source: Appendix Q · p. 25

Appendix R. Limitations and Broader Impacts

  • The benchmark covers ten curated tools and single-session tasks capped at 50 steps, excluding platforms such as LabView and longer multisession or multi-agent pipelines.It also uses English-only prompts, and visually rendered crystal structures can create SSIM-threshold edge cases due to GPU-dependent anti-aliasing.Source: Appendix R · p. 25
  • The paper recommends human review before agent-assisted results inform publication or engineering decisions because incorrect characterization settings could propagate errors into published datasets.Source: Appendix R · p. 25
Loading 2609.37053v1…