The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use

• Claude 3.5 Computer Use offers end-to-end GUI interaction from natural-language instructions and visual GUI state without external plans or GUI parsing.Source: §1 · p. 2
• The evaluation isolates planning, action, and critic capabilities: generating executable plans, grounding GUI elements, and adapting to action outcomes.Source: §1 · p. 2
• The case study covers desktop automation across web search, professional software, and games, reflecting varied user needs.Source: §1 · p. 2
• The authors provide Computer Use OOTB, an out-of-the-box cross-platform framework for easy deployment and benchmarking of API-based GUI automation models.Source: §1 · p. 2
• Human evaluations and analyses characterize both the advancements and limitations of the newly released API-based GUI automation model.Source: §1 · pp. 2–3

• Large vision-language models combine visual encoders with language models, while grounding studies target object localization such as bounding-box prediction [5] [6] [7] [8] [9].Source: §2 · p. 3
• GUI agents have expanded from web navigation to general automation by interacting with devices through human-like mouse and keyboard actions.Source: §2 · p. 3
• Researchers have pursued models that interpret human intentions and predict GUI actions as function calls, but available agent data remains limited and diverse.Source: §2 · p. 3
• SearchAgent adds inference-time search to improve multi-step reasoning and planning in interactive web environments.Source: §2 · p. 3

• The authors analyze Claude Computer Use from multiple perspectives, emphasizing both the underlying model and its tools.Source: §3 · p. 3

3.1 Model Design

• Claude Computer Use maps a natural-language instruction to a sequence of desktop actions while repeatedly observing GUI states and reflecting on outcomes.Source: §3.1 · p. 3

3.1.1 System Prompt

• The system prompt denotes environment-specific variables using full capital letters enclosed in square brackets.Source: §3.1.1 · p. 4

System Prompt

• The system exposes GUI, file-editing, and Bash tools within a sandboxed computing environment, with function results returned after invocation.The environment supports chained function calls and reports errors or outputs in subsequent result blocks.Source: §3.1.1 · pp. 4–5
• The system runs on an Ubuntu aarch64 virtual machine with Firefox, apt, pip, and GUI applications launchable through DISPLAY=:1.The current date and screen configuration are provided as environment parameters.Source: §3.1.1 · p. 5

3.1.2 State Observation

• Claude Computer Use observes the environment solely through real-time screenshots, without metadata or HTML, enabling general interaction with dynamic and closed-source GUI software.This vision-only approach imitates human desktop interaction and avoids dependence on software APIs for perception.Source: §3.1.2 · p. 5

3.1.3 Reasoning Paradigm

• Claude Computer Use follows a reasoning-acting paradigm that observes the GUI before acting and selectively observes only when necessary, reducing costs and accelerating execution.It can recognize when user requirements are fulfilled and stop without unnecessary actions.Source: §3.1.3 · p. 6

3.1.4 Tool Use

• Computer Tools provide keyboard, text-entry, mouse, drag, and screenshot actions for operating a desktop GUI through pixel coordinates.The system requires screenshot-based coordinate checking and cursor adjustment when clicks fail or applications take time to load.Source: §3.1.4 · pp. 6–7
• Editor Tools support persistent file viewing, creation, string replacement, insertion, and undo operations using absolute paths and line ranges.String replacement requires an exact, unique match, while viewing can target specified file-line ranges.Source: §3.1.4 · pp. 7–8
• Bash Tools execute shell commands with persistent state and access to mirrored Linux and Python packages through apt and pip.The Bash interface does not require XML-escaping command contents.Source: §3.1.4 · p. 9
• The tool-use guidance recommends inspecting targeted file ranges with sed rather than broadly reading files.It gives `sed -n 10,25p /path/to/the/file` as an example.Source: §3.1.4 · p. 9
• The prompt advises avoiding commands that may generate very large amounts of output.Source: §3.1.4 · p. 9
• Long-lived commands should run in the background, such as `sleep 10 &` or a background server.Source: §3.1.4 · p. 9

3.1.5 GUI Action space

• The GUI action space includes raw mouse and keyboard operations, with coordinate actions targeting pixel positions in observed screenshots.Actions are represented as action_type(arguments).Source: §3.1.5 · p. 9
• Supported GUI operations include mouse movement and clicks, typing, keyboard hotkeys, drag-and-drop, and screenshots.Examples include `mouse_move`, `left_click`, `type`, `key`, `left_click_drag`, and `screenshot`.Source: §3.1.5 · pp. 9–10

3.1.6 History Visual Context Maintenance

• Claude Computer Use retains an extensive history of screenshots accumulated during task operations.Source: §3.1.6 · p. 10
• Retained historical screenshots preserve full visual information along the trace, enhancing informed decisions as an episode unfolds.Source: §3.1.6 · p. 10

3.2.1 Out-of-the-Box Agent Framework

• Computer Use Out-of-the-Box enables local GUI-agent deployment on both Windows and macOS without Docker.It addresses the limitation of Anthropic’s demonstration codebase, which supports only Docker Linux environments.Source: §3.2.1 · p. 10

System Config.

• Testing uses Windows and macOS through Computer Use Out-of-the-Box at resolutions of (1366, 768) and (1344, 756), respectively.Source: §4.1 · p. 10

Human Review and Evaluation

• Human evaluators continuously monitor computer-use interactions and classify each completed task as Success or Failed.They also manually inspect the final state after task completion.Source: §4.1 · p. 10

Case Study Scope

• 20 tasks across 12 software or websites cover web search, workflow, office productivity, and video games.The cases were selected to represent a broad spectrum of desktop tasks across operating systems.Source: §4.1 · p. 10

Case Study Overview

• Table 1 summarizes the case studies by domain and indicates whether each task succeeded or failed.The table also links readers to the corresponding task sections.Source: §4.1 · p. 10, Table 1 · p. 11

4.2 Case Study: Web Search

• Web-search tasks require planning, robust grounding, and critique because dynamic pages contain many interactive elements and may require forward or backward navigation.The evaluation targets multi-step execution and adaptation to unexpected interface behavior in real websites.Source: §4.2 · p. 11

4.2.1 Find ANC Headphones Under Budget $100 on Amazon

• The Amazon task shows Claude planning a budget-constrained headphone search, locating visual controls, and verifying that the item was added to the cart.The model extracts a query from the instruction, filters by budget, grounds the search box visually, and checks the post-action screenshot.Source: §4.2.1 · p. 12

4.2.2 Browse Apple Official Site for Display with Accessories

• The Apple task demonstrates efficient search-based planning, visual interaction with varied elements, and final verification while handling a changing AppleCare window.The model selects search instead of menu traversal, interacts with text, buttons, and hyperlinked images, and lists the accessories after a final screenshot.Source: §4.2.2 · pp. 12, 14

4.2.3 Fox Sports Subscription (Failed)

• The Fox Sports case still shows dynamic replanning and situational awareness when the model encounters interface changes and authentication barriers.Its critic sequence explores alternative routes even though the final result is incorrect.Source: §4.2.3 · p. 14
• The Fox Sports task fails when Claude shifts from searching the expanded sports navigation to the Account tab instead of scrolling to find Formula 1.The detour triggers login requirements and exposes a limitation in scrolling-based navigation and contextually aware visual search.Source: §4.2.3 · pp. 14–15

4.3 Case Study: Workflow

• The workflow evaluation spans music management, Amazon-to-Excel transfer, spreadsheet export to local Excel, and App Store installation tasks.These cases cover multi-application interactions and multi-step objectives across web, productivity, and desktop software.Source: §4.3 · p. 15, §4.3.1 · p. 15, §4.3.2 · p. 17
• Together, the cases test whether the model can maintain context, ground actions in changing interfaces, and adjust plans across application boundaries.The workflow setting emphasizes consistency during data transfer, interface changes, unexpected behavior, and intermediate-state errors.Source: §4.3 · p. 15

4.3.1 Find Latest & Local Trending Music and Add to Playlist

• The music task requires locating songs in Apple Music and adding selections from Latest Songs and Singapore Top 100 to Sonnet’s Selection.The model navigates tabs and sections, uses Page Down with repeated screenshots, and repeats the playlist-addition procedure.Source: §4.3.1 · pp. 15–16
• The model repeatedly verifies Apple Music state and screenshots its position during scrolling before confirming both playlist additions.This critic loop checks tabs, menus, playlist options, navigation progress, and final completion.Source: §4.3.1 · pp. 16–17, Figure 5 · p. 16

4.3.2 Search for Products on Amazon and Record Prices in Excel

• The Amazon-to-Excel task transfers the first two monitor results into labeled worksheet cells, recording both products at $129.99.The sequence covers query entry, application switching, headers, product names, and prices.Source: §4.3.2 · pp. 17–18, Figure 6 · p. 17
• Visual checks before and after Excel entry verify application readiness and confirm that both products and headers were recorded correctly.The critic stage uses screenshots to reduce errors during cross-application data transfer.Source: §4.3.2 · p. 18

4.3.3 Export and Download Online Document to Open Locally

• The spreadsheet-export task downloads a Google Spreadsheet as Microsoft Excel (.xlsx), then opens the downloaded file locally in Excel.The plan crosses browser and desktop environments, while screenshots verify menu visibility, download completion, and file opening.Source: §4.3.3 · pp. 18–19, Figure 7 · p. 19

4.3.4 Install App from App Store and Report Storage Usage

• The App Store task searches for GoodNotes, reports its 1.39 GB size, and initiates installation until user authentication is required.The model stops at the intervention boundary rather than claiming that installation fully completed.Source: §4.3.4 · pp. 19–20, Figure 8 · p. 20

4.4 Case Study: Office Productivity Software

• Office applications lack the structured interfaces common on the web, requiring the model to perceive dense visual controls and execute precise actions across software versions and themes.Minor inaccuracies can cause unintended changes, making visual grounding, action precision, and layout adaptation central challenges.Source: §4.4 · pp. 20–21

4.4.1 Forward a Specific Email and CC Another Recipient

• The model successfully forwarded the specified Anthropic email, added recipient and CC addresses, sent it, and verified each step through screenshots.The systematic checks demonstrate self-monitoring during a professional email task.Source: §4.4.1 · pp. 21–22, Figure 9 · p. 21

4.4.2 Change Document Layout to A3 Size in Landscape Orientation

• The model correctly changed a Word document to A3 landscape and reformatted another document from one column to two columns, confirming both layout outcomes.These tasks used sequential navigation through Word’s Layout, Size, Orientation, and Columns controls.Source: §4.4.2 · pp. 22–23, §4.4.3 · p. 23

4.4.4 Update Name and Phone Number on Resume Template (Failed)

• The resume-editing task failed because imprecise selection changed only part of the name and phone number, while the model incorrectly judged both replacements complete.A double-click selected “Gardner” rather than the full name, and incomplete phone-number selection was not detected during verification.Source: §4.4.4 · p. 24, Figure 12 · p. 24

4.4.5 Gradient Fill Background

• In PowerPoint, the model recovered from a failed initial route to apply a gradient background and successfully added a “Triangle” title with a triangle shape.The gradient task demonstrates replanning through the Design tab when the expected Format Background pane was unavailable.Source: §4.4.5 · pp. 25–26, §4.4.6 · pp. 26–27

4.4.7 Insert Numbering Symbol (Failed)

• The numbering task failed after the model applied bullets, struggled to select all text and locate numbering controls, and did not reliably verify the final format.The case highlights limitations in precise selection, small-button localization, and final outcome checking.Source: Figure 15 · p. 27, §4.4.7 · p. 28

4.4.8 Find and Replacement in Worksheet

• Excel’s Find and Replace operation successfully changed “$” to “SGD” across the worksheet, with Excel confirming 190 replacements.The model used Ctrl+H, entered both strings, selected Replace All, and checked Excel’s confirmation message.Source: §4.4.8 · pp. 28–29

4.4.9 Insert a Sum Equation over Cells

• The SUM task selected C6:C15 instead of the requested ACTUAL-column range, excluding “Other Expenses,” and the model failed to detect the incorrect calculation.The model’s plan identified the correct column, but execution used an incorrect range and the available passage reports no successful final verification.Source: Figure 17 · p. 29, §4.4.9 · pp. 29–30
• The critic explained the SUM function but assumed the calculation was correct without confirming that the selected cells matched the user’s request.This indicates insufficient final confirmation of the range’s contextual correctness.Source: §4.4.9 · p. 30
• The model summed C6:C15 instead of the correct D6:D16 “ACTUAL” range, excluding the “Other Expenses” row and producing an incomplete calculation.Its step-by-step item breakdown did not detect the wrong column or missing cell.Source: §4.4.9 · p. 30
• The case demonstrates that improved self-critique and selection accuracy are needed for reliable Excel data processing.The limitation primarily concerns range selection and mathematical reasoning.Source: §4.4.9 · p. 30

4.5 Case Study: Video Games

• The study uses Hearthstone and Honkai: Star Rail to examine GUI automation in strategic, visually rich, and dynamically changing game environments.Hearthstone emphasizes deck building, resource management, and tactical decisions, while Honkai: Star Rail stresses visual grounding across intricate menus and unlabeled icons.Source: §4.5 · p. 30
• Overall, the examples show adaptation across complex game interfaces, but successful execution is supported by detailed instructions and carefully structured multi-step tasks.The cases demonstrate capability in planning, visual interaction, and critic behavior rather than unrestricted gameplay autonomy.Source: §4.5 · p. 30, §4.5.3 · p. 33, §4.5.2 · p. 33

4.5.1 Hearthstone: Create and Rename a New Deck for Battle

• The model creates and renames a Hearthstone Core Mage deck through eight interface actions, then verifies the new title and deck availability.It navigates class selection, preset selection, renaming, confirmation, and final verification successfully.Source: §4.5.1 · pp. 31–32

4.5.2 Hearthstone: Hero Power

• In Hearthstone, the model identifies a 2/1 enemy minion as the optimal Hero Power target with 3 mana, eliminates it, and ends the turn after checking the board.The task requires interpreting mana, health values, stylized visuals, and the resulting game state.Source: §4.5.2 · pp. 32–33

4.5.3 Honkai: Star Rail: Warp Automation

• The model successfully follows a detailed 10-warp sequence in Honkai: Star Rail, including selecting the banner, skipping animations, and closing the results screen.It repeatedly checks for the skip arrow and verifies acquired characters before exiting.Source: §4.5.3 · p. 33
• Detailed user instructions help the model automate the warp task in an unfamiliar game environment while maintaining alignment with the requested sequence.The authors present this as evidence of accurate multi-step following and repetitive-task automation.Source: §4.5.3 · p. 33

4.5.4 Honkai: Star Rail: Daily Mission Clean up Automation

• The model automates a Honkai: Star Rail daily mission by selecting the EXP Calyx, setting six attempts, starting auto-battle, monitoring progress, and exiting after completion.The long trajectory combines menu navigation, parameter setting, combat automation, and state monitoring.Source: §4.5.4 · pp. 33, 34, 36
• Periodic screenshots and visual checks let the model confirm battle stages and task completion during the daily-mission automation.The critic loop checks indicators such as “Battle Start” and the completion screen before proceeding.Source: §4.5.4 · p. 36

5.1 Error Categorization

• Planning errors reflect incorrect task interpretation or state understanding, while action errors reflect failures in interface understanding, spatial recognition, or precise GUI control.Source: §5.1 · p. 36
• Critic errors occur when the agent misjudges its actions or computer state and provides erroneous completion feedback.Source: §5.1 · p. 36

5.2 Toward Future GUI Agents

• Dynamic benchmarks should reflect changing software versions, screen resolutions, and interactive real-world complexity because static datasets limit assessment of agent adaptability.Source: §5.2 · p. 36
• Critic errors frequently arise when the model incorrectly assumes a task is complete, motivating stricter internal critic mechanisms beyond prompting alone.Source: §5.2 · p. 37
• Current agents diverge from nuanced human computer use in scrolling and rigorous browsing, partly because training data underrepresent variable, context-specific adaptations.Source: §5.2 · p. 37

• The case study examines Claude 3.5 Computer Use across web navigation, workflows, productivity software, and video games, emphasizing planning, action execution, and critic feedback.Source: §6 · p. 37
• Computer Use Out-of-the-Box aims to make API-based GUI agents easier to deploy and benchmark in real-world scenarios.Source: §6 · p. 37