OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software

1 Introduction

• Scientific computer use requires understanding specialized interfaces, maintaining state across long workflows, and producing scientifically valid artifacts.Scientific work spans simulation, visualization, data analysis, and experimental control, often with steep learning curves and heterogeneous data.Source: §1 · p. 2
• Existing evaluations leave gaps in scientist-prioritized task design, unified evaluation of heterogeneous scientific outputs, and separating harness effects from backbone-model performance.The harness issue matters because hybrid GUI and tool interfaces can materially change computer-use performance.Source: §1 · p. 2
• The strongest agent achieves about 70% average success and about 40% on the hardest set, despite OSWorld-Science’s challenging scientific tasks.The benchmark combines expert-designed tasks, heterogeneous-output evaluators, and an improved CUA harness.Source: §1 · p. 2

2 Related Work

• General computer-use benchmarks expanded from web navigation to desktop, Windows, and mobile workflows, with execution- or state-based evaluation.OSWorld-MCP also studies agents choosing between GUI actions and structured tools, treating the interaction layer and harness as experimental variables.Source: §2 · pp. 2–3
• Scientific agents support literature synthesis, domain-tool use, experimental planning, and research automation, but evaluations are often tied to one architecture, domain, or tool collection.Examples include PaperQA, ChemCrow, Coscientist, and The AI Scientist.Source: §2 · p. 3
• ScienceBoard and Terminal-Bench-Science evaluate scientific workflows in desktop and command-line settings, while SciAgentArena covers multi-step challenges across domains.OSWorld-Science is complementary: it uses practicing scientists’ importance and difficulty judgments, supports diverse visual and structured outputs, and analyzes and optimizes the harness.Source: §2 · p. 3

3 Tasks, Environments, and Harness in OSWorld-Science

• Tasks are goal-conditioned POMDPs, with task-specific evaluators assigning full, partial, or zero credit according to scientific requirements.The execution trajectory, task-relevant artifacts, and goal-specific evaluator determine the score.Source: §3 · pp. 3–4
• The modeled environment includes hidden application and scientific state, while observations expose interface-level information and rewards measure goal satisfaction.Scientific tasks can involve latent objects such as voxel coordinates, segmentation masks, transformation matrices, measurements, parameters, or generated files; evaluators inspect environment state and artifacts.Source: §3 · pp. 3–4
• Task selection spans seven scientific domains and three languages, guided by scientific value and a difficulty gradient that includes solvable and unsolved tasks.Tasks come from domain-expert proposals and an iterative AI-assisted co-design process based on anchor tasks such as software tutorials.Source: §3 · pp. 4–5
• The benchmark uses Ubuntu task-specific images for 17 scientific software packages, including both open- and closed-source applications.The setup tests software interaction with the base agent framework and secures licenses for closed-source software.Source: §3 · p. 5
• A shared OSWorld-based PromptAgent harness drives model comparisons with screenshots, recent interaction history, GUI and terminal control, fixed budgets, and trajectory logs.The harness adds adapters, loop control, and reliability measures while keeping message construction and prompting unchanged across models.Source: §3 · p. 5

Dataset overview

• 146 tasks span seven domains and 14 software configurations, with chemistry largest (43; 29.5%) and linguistics smallest (2; 1.4%).The distribution reflects the current collection effort; the authors motivate reporting aggregate and domain-macro-averaged performance.Source: Figure 2 · p. 5, §4 · p. 6

Overall performance

• 73.7% mean task-specific score at $6.5 per task gives Claude Fable 5.1 the highest overall result among 12 VLMs.Scores include partial credit, and VOID runs receive zero.Source: Figure 3 · p. 6, §4 · p. 6

Domain results

• Model rankings shift sharply by domain: scores range from approximately 66% to 97% in statistics, while eight chemistry models score approximately 6% or less.Astra reaches about 6% in chemistry and 100% on the two linguistics tasks.Source: §4 · p. 7

Trajectory analysis

• 667 of 1,100 non-full-credit runs ended without the graded artifact, and interaction measures did not predict full credit after model and task controls.Only 402 of 1,530 runs received full credit; within-task associations between interaction measures and success vanished among runs of the same model in the same domain.Source: §4 · p. 7, Figure 4 · p. 8

Ablation studies

• On 23 QuPath tasks, effort and history-window changes shifted scores while prompt language changed Opus 5’s interaction style; no cell differed significantly from default after correction.Opus 5 rose from 44.1% at medium effort to 59.7% at max; history window w = 10 scored 57.0%, versus 35.2% at w = 1.Source: §4 · pp. 7–8, Figure 5 · p. 8

Selective case study

• Nearly equal-length chemistry workflows diverged: Lenacapavir earned 1.0 after source-grounded checks, whereas Archangiumide scored 0.0 after self-consistency checks preserved a stereochemical error.The successful run validated all 17 rows; the failed run had a correct graph and formula but incorrect stereochemistry and InChIKey.Source: Figure 6 · p. 9, §4 · p. 9

• OSWorld-Science combines task design, a CUA harness, and evaluation metrics to assess agents operating software to solve scientific problems.Source: §5 · p. 10
• The experiments find that state-of-the-art VLM-based agents still have room to improve, particularly in scientific problem understanding and learning software operation.Source: §5 · p. 10
• The analysis identifies common operational errors that are difficult to resolve through conventional approaches such as increasing reasoning effort or expanding the context window.Source: §5 · p. 10

• The question sets and Docker images are available through the OSWorld-Science dataset on Hugging Face.Source: §6 · p. 10
• The project code and project website are also provided through GitHub and Hugging Face, respectively.Source: §6 · p. 10
• Because some evaluated software requires a license, the authors do not plan to publish all trajectories in this version.Source: §6 · p. 10

• Experiments follow OSWorld’s design, in which models choose actions by analyzing screenshots and trajectories.Source: A Experimental details · p. 14
• All VLMs except Kimi-K3 use the modified mmagent; Kimi-K3 uses its officially provided design because of model adaptation requirements.Source: A Experimental details · p. 14
• Single-solution tasks use exact-match scoring, sometimes with partial credit for steps, while simulation tasks are scored by discrepancies from actual parameters.Source: A Experimental details · p. 14

• The appendix compares overall task score with output-token use, API cost, and domain-level performance and cost across the supplied evaluation plots.The plots cover 146 evaluated tasks across eight domains; these counts refer to plotted results, not the full task inventory.Source: B Additional performance and cost… · p. 15, Figure 7 · p. 15, Figure 8 · p. 16
• Scores are task-weighted means over tasks run by each model, with VOID runs assigned zero; token means cover only tasks with recorded usage and may have different coverage.Source: B Additional performance and cost… · p. 15
• Figure 7 plots mean task score against mean output tokens per task, while Figures 8 and 9 separately rank API cost and task score.Output-token consumption and API dollar cost are distinct efficiency measures.Source: Figure 7 · p. 15, B Additional performance and cost… · p. 15, Figure 8 · p. 16
• The multilingual table reports mean partial-credit scores and language shares among replies containing prose, with Opus 5 shares based on only 28–65 replies per cell.Source: Table 1 · p. 19
• The QuPath ablation table defines partial-credit and binary scores, mean steps, and mean output tokens, with VOID runs counted as zero under both score metrics.Source: Table 2 · p. 20

Steps and outcomes

• Long GUI-heavy tasks often use all 100 steps and remain near zero, while finishing within 20 steps corresponds to 57% average for Opus 5 versus 3% for 100-step runs.For Sonnet 5, 60% of runs use all 100 steps and average 3%, compared with 32% for earlier-finishing runs; some long tasks switch between VOID and full credit depending on finishing within the limit.Source: C Additional ablation analyses · p. 21

Reasoning effort

• Max reasoning effort newly solves five Opus 5 tasks but regresses on five, while Sonnet 5 spends more tokens with little outcome change.For Opus 5, effort increases thinking share and GUI inspection, and can turn a short script into 100-step exploration; Sonnet 5 solves only 2–4 tasks across settings.Source: C Additional ablation analyses · p. 21

History window

• For Opus 5, history window w = 10 reduces mean steps to 22 with no VOID runs, while w = 15 matches its steps and tokens but loses ground on five tasks.The two long GUI-heavy tasks that are VOID at w ≤5 finish in 43–57 steps at w ≥10.Source: C Additional ablation analyses · p. 21

Task language

• Spanish, French, and Thai prompts shift Opus 5 toward GUI interaction and lengthen trajectories, while Chinese and Japanese preserve its scripting share and trajectory length.The two long GUI-heavy tasks flip between VOID and full credit across languages; some English-solved tasks fail under Chinese in similarly short runs with wrong results.Source: C Additional ablation analyses · p. 21

Outcome labels

• Outcome labels are determined by the graded artifact and ordered checks, so a well-formed but incorrect artifact remains wrong content even when the run also exhausts its budget.The label scheme distinguishes full credit, harness or environment endings, no deliverable, unresolved records, invalid artifacts, and wrong content.Source: §D.1 · p. 25

Run outcomes

• Among the 667 runs without a deliverable, 613 exhausted their step or time budget, 47 stopped of their own accord, and 7 were stopped after consecutive replies without an executable action.Source: §D.1 · p. 25

Matched comparisons

• None of the five fixed-effects interaction measures is significant after Bonferroni correction; GUI share has an uncorrected coefficient of −0.067 (SE 0.033, p = 0.05).Models use N = 1,496 runs, except thinking with 951 runs on 85 tasks.Source: §D.1 · p. 26

Further observations

• In 72 of 77 GPT-model chemistry placeholder runs, agents exhaust the budget before filling the skeleton; the two Qwen models repeat 21–25% of replies on average.GPT-5.6 sol, terra, and luna press a key in 53–63% of steps, while MiniMax M3 narrates in 79% of replies.Source: §D.1 · p. 27

Cross-software and within-group comparisons

• Mean cross-software CLI-share rank correlation is 0.43, while within model-domain groups every median outcome difference lies within five percentage points of zero.The fixed-effects GUI-share estimate leans negative, with uncorrected p = 0.05.Source: §D.1 · p. 27

Efficiency of successful runs

• Full-credit runs take a median of 19 steps, ranging from 12–14 for four models to 27–60 for Qwen3.7-Plus, Qwen-CUA, Sonnet 5, Kimi K3, and MiniMax M3.No model receives full credit on 40 of the 128 tasks, while every model solves two tasks.Source: §D.1 · p. 27

Geographic questions

• 13 of 66 runs succeed, and the mean partial-credit score is 32.5%; 28 failed runs deliver incorrect results, 21 are VOID, and four stop early.Table 13 reports every run by task, with success and failure-cause labels.Source: §D.2 · p. 27, Table 13 · p. 31

Successful runs

• Eleven of the 13 successful runs use scripts, and all seven successful GEO 001 runs read back their result table before stopping.On GEO 005, GPT-6-Astra scores 0.98 in four steps, Opus 5 scores 0.90 after 88 steps, and Fable 5.1 scores 0.76 with part of the network missing.Source: §D.2 · p. 28, Figure 12 · p. 30

Interaction-layer failures

• In 23 of 53 failed runs, intended actions never take effect; four backbones miss targets because they ignore the specified 1920 × 1080 resolution.Those four backbones account for 18 such runs; neither GPT-6-Astra nor Fable 5.1 fails at this stage.Source: §D.2 · p. 28

Output verification

• Some output checks miss substantive errors: Fable 5.1 reports a 2.8-pixel fit residual on four control points, while error at 104 independent check points is 216 pixels.None of the eight runs delivering GEO 004 footprints checks whether all buildings are covered.Source: §D.2 · pp. 28–29

Method and completion failures

• Among runs reaching the desktop, inadequate methods account for 12 failures and incomplete work for 13; six GEO 003 results use too few control points for image-wide deformation.On GEO 006, Fable 5.1 correctly grades 20 of 30 buildings after cropping before-and-after image pairs, the task’s highest result.Source: §D.2 · p. 29

Scope of the analysis

• The analysis uses one run per backbone and task, so its 66 trajectory counts have no variance estimate, and each failed run receives one primary cause.Source: §D.2 · p. 29

The terminal as the default instrument

• 214 of 240 runs never opened SAS Studio, RStudio, or a GUI editor, and the two scientific applications were used in only 10 runs.All runs began with the SAS sign-in page visible, but only nine reached a live SAS Studio session; just two models reached it.Source: §D.3 · p. 32

Classifying trajectories and the limits of the classifier

• The application-use classifier relies partly on model narration because coordinate-based clicks leave no textual trace of which application was controlled.Executed code established application use in 22 of 26 classified runs; the other four were adjudicated manually, and two independent audits agreed on nine live SAS sessions.Source: §D.3 · p. 32

The census

• 183 of 240 runs used only a shell, 31 also opened a viewer, and 26 drove an application: SAS Studio in 9 runs, VS Code in 9, gedit in 9, and RStudio in 1.Agents opened a terminal in 209 runs; the median did so in its first step.Source: §D.3 · p. 32, Table 15 · p. 34

Instruction wording and tool choice

• When SAS was preferred but alternatives were allowed, agents treated that permission as decisive; all 24 runs on two tasks restricting analysis to SAS or R used R.None interacted with the SAS page already visible, and no run argued that R or Python was intrinsically the better statistical instrument.Source: §D.3 · p. 33

What the interaction mode costs

• Within tasks, application-mode use was associated with 29 additional steps and a −0.139 score difference, but GUI-editor use—not domain-application use—accounted for the detectable score penalty.Domain applications had an estimated within-task score difference of −0.069 (p = 0.158); GUI editors had a 0.228 lower score (p = 0.012). These comparisons are not causal, and the domain-application score comparison was underpowered.Source: §D.3 · p. 34, Figure 15 · p. 35, Table 16 · p. 35

Case study: interface overhead and task completion

• On stat group tests, gpt-6-astra completed in R in 12 steps and scored 1.000, whereas kimi-k3 obtained correct SAS statistics but exhausted 85 steps without an output file and scored 0.080.The contrast illustrates how interface overhead can separate statistical correctness from successful delivery.Source: §D.3 · pp. 35–36, Figure 16 · p. 36

D.4 Linguistics phonetic annotation: Praat interface operation and acoustic landmark selection

• 43 of 60 required Praat boundaries were absent; of the 17 newly placed in saved TextGrids, 12 fell within tolerance.Fifteen of 24 runs saved no TextGrid, and another saved one without inserting a measurement boundary.Source: §D.4 · p. 37
• Consistent error directions and glottal-period-scale offsets point to landmark choice as well as interface precision; a one-cycle error exceeds the 2 ms tolerance for bat and bit.gpt-6-astra placed two boundaries within 0.09 and 0.07 ms of their references, while claude-sonnet-5 placed the neg boundary 51.7 ms late.Source: §D.4 · pp. 38–40

D.5 EEG: Trial Alignment and Event-Pairing Audit

• On RT–P3, Opus 5 retained 72 clean responded trials, split evenly between fast and slow groups, and estimated a slow–fast difference of 0.93 µV with a 95% bootstrap interval spanning zero.On the separate forensic task, six reused response events affected 12 stimulus rows, but the model reported six surplus assignments instead.Source: §D.5 · p. 41

D.6 Structural biology and NMR: application gates and coordinate frames

• Of 84 scored structural-biology and spectroscopy runs, 57 succeeded; all backbones solved align, and the extraction and mutation tasks were solved in 34 of 40 scored runs.Synergy and NMR, the two tasks hinging on GUI dialogs, accounted for 18 of 27 failures.Source: §D.6 · p. 42
• Coordinate-frame mismatches affected 11 of 13 interaction-layer failures, while a SynergyFinder server outage left three runs unscored and illustrates the availability dependence of live third-party services.The mismatched frames fit most clicks when rescaled; a locally hosted SynergyFinder instance would remove the service-availability dependence.Source: §D.6 · pp. 42–43
• Five runs opened the NMR spectrum, but seven never reached it; four runs received partial credit, with the merged 7.30–7.45 ppm multiplet costing 7.9 rubric points in each.The staged installation of Mnova did not take effect in these runs, although the instructions stated that it was installed.Source: §D.6 · p. 43, Figure 20 · p. 45
• The structural-biology analysis covers 84 scored trajectories with one run per backbone and task, so its counts carry no variance estimate.Coordinate frames were inferred from where clicks landed relative to the controls agents described; this fit most, but not all, clicks.Source: §D.6 · pp. 43–44

E Statistics analysis

• The task-set figure reports 266 non-exclusive capability annotations and screenshot-based visual observation for all 145 tasks; the accompanying figure also presents CLI-support proportions.The supplied figure captions do not state the CLI proportion values.Source: Figure 21 · p. 45, E Statistics analysis · p. 45

E.1 Medicine Analysis

• Ten of twelve models deliver nothing on QP 004 T1, while GPT-6 Astra and Fable 5.1 both score 1.0.The task requires installing QuPath’s Interactive Image Alignment extension before registering the attention map.Source: §E.1 · p. 45
• GPT-6 Astra registers the extension through QuPath, verifies its Analyze entry, and exports the required image in 31 steps.The run scores 1.0 and uses the command finder after menu navigation stalls.Source: §E.1 · pp. 45–46
• Claude Opus 5 copies the extension into a guessed directory, but QuPath never registers it; the run is VOID at the 100-step cap.The Analyze menu remains empty from step 19 to step 98 despite nine relaunches.Source: §E.1 · p. 46
• QuPath’s empty Analyze menu was visible 79 steps before the budget ran out, but Opus 5 repeated its install-and-relaunch cycle.The comparison attributes the difference to verifying the prerequisite through the application, not trajectory length or effort.Source: Figure 22 · p. 46, §E.1 · p. 46

F Biology Analysis

• Seven of twelve models never open the 6-bromoindole spectrum; all five that pass Mnova’s licensing gate deliver an assignment.The task requires accepting the license, choosing Install, and importing a license file before analysis.Source: F Biology Analysis · p. 47
• Claude Opus 5 scores 0.75 in 25 steps after replacing its recalled assignment with Mnova’s spectrum analysis.The overlapping C2-H/C4-H pair remains grouped as one two-proton signal, limiting the score.Source: F Biology Analysis · p. 47
• GPT-5.6-luna misses Mnova’s Install button repeatedly and is scored VOID at the 100-step cap.From steps 50 to 97, 39 clicks land beside the wizard; the unchanged screen provides evidence that the clicks are not working.Source: F Biology Analysis · p. 47
• The Mnova case identifies coordinate-frame alignment as the interaction bottleneck, while every run that opens the spectrum retains the overlapping-pair grouping.The authors recommend checking clicks against the screen frame and staging analysis tasks with the application installed and licensed.Source: F Biology Analysis · p. 47

F.1 Physics Analysis I: CIAO in Astronomy

• 14 of 36 runs succeed, with a mean partial-credit score of 45.1%; failed runs include 13 incorrect answers, 8 VOID runs, and 1 empty completion.The evaluation uses one run per backbone and task, with a limit of 100 interaction steps, extended to 150 for eight backbones on lightcurve.Source: §F.1 · p. 48
• Successful runs take final values from full-precision CIAO output, compute them from data, or check them directly against the image; trajectory length alone says little about outcome.Successful runs take 6 to 77 steps, while failed runs that deliver an answer take 5 to 100.Source: §F.1 · p. 48
• In 8 of 22 failed runs, actions miss their intended targets; scaled coordinate frames cause clicks to land away from the intended controls.The affected actions include clicks on terminals, menu items, and dialog buttons, and seven of these runs are VOID.Source: §F.1 · pp. 48–49
• On netcounts, the correct value is never visible in DS9’s result window, so successful runs compute it or read the output file instead.When the annulus is tagged as background, the window opens scrolled past NET COUNTS; successful answers are within 0.0096 counts of the reference.Source: §F.1 · p. 49
• On lightcurve, both times are almost always right, but the rates are right in 6 of 11 runs; successful rate calculations preserve CIAO’s exposure and background normalization.Wrong rates arise when runs divide by nominal bin width or apply background scaling twice.Source: §F.1 · p. 49
• On streak, vertical-line assumptions place points too far from the tilted streak, while checking the magnifier or fitting the event data supports better placement.The streak is 6.1° from vertical; Fable 5.1 checks the magnifier, and Opus 5 and Kimi-K3 derive the line from events.Source: §F.1 · pp. 49–50
• GPT-6 Astra scores 1.0 on netcounts by verifying the regions, reading the full-precision output file, and checking the saved workbook value.GPT-5.6-terra enters 1410.7212 by subtracting BG ERR from raw counts, then validates the saved entry against itself and scores 0.0.Source: §F.1 · pp. 50–51
• One run per backbone and task means the 36 trajectory counts have no variance estimate.Each failed run is assigned a single primary cause.Source: §F.1 · p. 51

F.2 Physics Analysis II: Ansys Fluent in FLUID DYNAMICS

• 66 of 168 ANSYS Fluent runs succeed, with a mean partial-credit score of 40.0%; 85 of the 102 runs without full credit are VOID.The evaluation covers 14 tasks, one run per backbone and task, with a 100-step limit.Source: §F.2 · p. 53
• Four backbones account for 48 of 66 successes, and most successful runs combine Fluent’s graphical interface with console commands.Setting up a case is solved in 19 of 60 runs, compared with 27 of 60 for reading one out.Source: §F.2 · p. 53
• In 46 of 102 runs, intended actions never take effect; four backbones use scaled coordinates and score 0 in all 56 of their runs.Other interaction failures include keystrokes reaching the wrong widget or being lost in the Export dialog.Source: §F.2 · p. 53
• Thirty-eight runs exhaust the budget without a gradable report, often after repeated setup checks or re-entry of settings already applied.Twelve runs run out while setting up a task without iterating the solver; waiting for the solver does not by itself exhaust the budget.Source: §F.2 · pp. 53–54
• All ten reports scoring below 1 come from GPT-5.6 variants, and at least 15 runs with correct values first produce missing, empty, or invalid JSON.Method errors and failed self-checks often involve single-dialog setup slips or physically implausible displayed values that the agent does not use to find the error.Source: §F.2 · p. 54
• On the adjoint task, an observable without a selected wall evaluates to 0 N and makes Fluent report convergence after one iteration.Kimi-K3 corrects the zone after seeing 0 N and scores 1.0; GPT-5.6-terra retains the setup belief despite contradictory readings and scores 0.33.Source: §F.2 · pp. 54–55
• The check helps only when its result can overturn the assumed setup; the authors recommend treating physically implausible values as evidence against it.They also recommend benchmarks include quantities, such as iteration count, that expose silent setup errors.Source: §F.2 · p. 55
• The analysis has one run per backbone and task, so its counts describe 168 trajectories without a variance estimate.The authors caution that single outcomes of long runs should be read with care; Kimi-K3’s trajectory shapes are only partly comparable because it uses its own tool-calling loop.Source: §F.2 · pp. 55–56

Adjoint-task trajectory contrast

• On FT-008a, Kimi-K3 corrected its 0 N drag reading by selecting the wall zone, while GPT-5.6-terra trusted the stored report and ran a zero-source adjoint.Kimi-K3 scored 1.0 after 86 steps; GPT-5.6-terra scored 0.33 after 100 steps.Source: Figure 25 · p. 57

Failures before solving

• 49 of 91 failed runs (54%) never reached a solver iteration on at least one required mesh.Five backbones accounted for 40 of these 49 early failures, while four other backbones had no failures in causes A–D.Source: §F.3 · pp. 57–58

Solving a different problem

• Twenty runs used settings that contradicted the task statement; five diverged, while the other 15 converged to a different solution.In eight cases, replaying the agent’s case with a single change restored a score of 1.Source: §F.3 · p. 58

Correct cases that missed the grader

• Eighteen failures involved a correct case whose solution was unfinished or lost at export; a single-change replay restored a score of 1 in 12.All six lossy-export failures built their matrices from foamToVTK output; five involved Float32 arrays printed with six digits.Source: §F.3 · p. 58

Case study: correct physics, lost digits

• On OF-012, GPT-5.6-sol passed all four gates after exporting 12-digit native fields and checking row counts, headers, and physical values.The run earned 1.0 after 35 steps; it rejected Float32 VTK arrays as insufficiently precise.Source: Figure 26 · p. 59, §F.3 · pp. 59–60
• On OF-004, GPT-5.6-sol earned 0.125 after exporting six-digit arrays, despite solving the case; a replay scored 1 with 12-digit fields.The coarse-mesh bulk velocity differed from Ub in the eighth significant figure, but the run declared DONE without checking that value against the required precision.Source: Figure 26 · p. 59, §F.3 · pp. 59–60

Overall outcomes

• 77 of 168 runs succeeded, with a mean partial-credit score of 47.3%.The analysis covers 14 OpenFOAM tasks, with one run per backbone and task; these counts have no variance estimate.Source: §F.3 · pp. 56, 60, Table 25 · p. 62