ScreenAgent: A Vision Language Model-driven Computer Control Agent

• Current VLM agents typically cannot generate and execute continuous manipulative commands in real computer or mobile environments.Source: §1 · p. 2
• ScreenAgent uses planning, execution, and reflection to assess actions, provide feedback, and sustain a continuous thinking chain.Source: §1 · p. 2
• The work introduces a VNC-based environment for direct mouse-and-keyboard interaction with real computer screens.Source: §1 · p. 2
• The ScreenAgent dataset contains action sequences for generic Linux and Windows desktop tasks and supports fine-grained capability evaluation.Source: §1 · p. 2
• ScreenAgent achieved results comparable to GPT-4V across aspects while providing more precise positioning, addressing GPT-4V’s positioning weakness.Source: §1 · p. 2

2.1 Multimodal Large Language Models

• GPT-4V, LLaVA-1.5, and CogVLM provide multimodal or visual-language capabilities, while Monkey enhances input-resolution capability and Fuyu-8B uses a pure decoder Transformer without an image encoder.Source: §2.1 · p. 2

2.2 Computer Control Environment & Dataset

• Existing computer-control resources mainly target simulated web environments, webpage understanding, or Android interaction rather than universal desktop mouse-and-keyboard control.Source: §2.2 · p. 2

2.3 Large Language Model-Driven Agents

• ScreenAgent’s planning, acting, and reflecting pipeline decomposes tasks, generates function-call operations, and decides whether to continue, retry, or reformulate the plan.Source: Figure 2 · p. 3
• LLM-driven agents have been applied to web browsing, tool use, lifelong Minecraft learning, memory reflection, process automation, GUI navigation, and smartphone operation.Source: §2.3 · p. 3

3.1 Computer Control Environment

• The environment connects to a desktop operating system through VNC and sends mouse and keyboard events to the controlled computer.Source: §3.1 · p. 3
• Screenshots define the state space, while JSON-style function calls define executable actions parsed and sent through the environment.Source: §3.1 · p. 3

3.2 Control Pipeline

• The planning phase decomposes a complex task into subtasks using the current screenshot and the agent’s common-sense and computer knowledge.Source: §3.2 · p. 3
• The annotation process pairs GPT-4V responses with corrected golden responses after executable actions are run on a real computer.Source: Figure 3 · p. 4
• The acting phase converts screenshot-based mouse and keyboard decisions into VNC device actions and captures the resulting screen for the next step.Source: §3.2 · p. 4
• The reflecting phase evaluates the after-action screen and chooses whether to retry, continue, or adjust the plan for unpredictable circumstances.Source: §3.2 · p. 4

Dataset Statistics

• The dataset statistics report task-type composition and distributions of action types and chosen response-token counts.Source: Figure 4 · p. 4, Figure 5 · p. 4

ScreenAgent Dataset

• 273 task sessions span 39 sub-task categories across 6 themes, with 203 training sessions and 70 testing sessions collected from Linux and Windows.The dataset includes 3005 training screenshots and 898 testing screenshots.Source: §4 · p. 4
• ScreenAgent targets broader computer use than datasets focused mainly on web browsing or Android by covering office work, booking, retrieval, games, entertainment, programming, and system operations.Source: §4 · p. 4

Evaluation

• Table 2 evaluates successful function-call proportions on the ScreenAgent test set, while Table 3 compares fine-grained scores for successfully matched actions.Source: Table 2 · p. 5, Table 3 · p. 5

CC-Score

• CC-Score measures control-action sequence similarity through action ordering, attribute consistency, and text similarity across mouse, text, and keyboard actions.Mouse actions consider operation type, button, and feasible-coordinate-box inclusion; text and keyboard actions use attribute consistency and BLEU.Source: §4 · pp. 4–5

5.1 Evaluation Results on ScreenAgent Test-Set

• GPT-4V and LLaVA-1.5 achieved higher function-call success rates, while CogAgent variants often ignored the required JSON format and became unable to interact with the environment.Using GPT-3.5 to extract JSON-style calls from CogAgent-Chat still left its scores significantly below LLaVA-1.5 and GPT-4V.Source: §5.1 · pp. 5–6
• GPT-4V achieved a 0.98 action type prediction F1 score, remaining the strongest performer for selecting mouse and keyboard actions.This evaluation concerns predicted attribute values within successfully parsed function calls.Source: §5.1 · p. 6
• Precise positioning remains difficult: GPT-4V refused precise coordinates, and two open-source models also failed to output correct coordinates under the pipeline prompt.The paper identifies positioning as crucial for computer manipulation.Source: §5.1 · p. 6
• All models struggled with reflection, with GPT-4V reaching only a 0.60 F1 score for deciding whether subtasks were complete or needed adjustment.The authors state that human intervention remains necessary during task execution.Source: §5.1 · p. 6

5.2 Fine-tuning Training

• ScreenAgent achieved GPT-4V-level instruction following and function-call performance, comparable action-level scores, and substantially more precise mouse clicking after fine-tuning.The results also report a significant planning gap relative to GPT-4V.Source: §5.2 · p. 6

5.3 Case Study

• In case studies, ScreenAgent produced concise plans, correctly positioned clicks where other agents failed, and retried actions after reflection when needed.The comparison covers planning, acting, and reflecting phases across example tasks.Source: §5.3 · p. 6

6 Conclusion

• ScreenAgent combines a real-computer screen-control environment, planning-acting-reflecting pipeline, broad everyday-work dataset, and fine-grained action- and task-level evaluation.The system uses direct mouse and keyboard control commands.Source: §6 · p. 6
• ScreenAgent achieved scores comparable to GPT-4V while providing more accurate UI element localization.The conclusion frames this result as support for further research on more powerful and generalized agents.Source: §6 · p. 6
• The model processes only single-frame images rather than videos or multi-frame images, and its language capability depends on the foundation language model.The authors also report limited GPT-4V support for non-English screen text.Source: §6 · p. 7

Ethical Statement

• The authors note societal risks including employment impacts, fraud, abuse, privacy issues, and misoperation.They specifically mention possible CAPTCHA bypass, misinformation, and illegal activity.Source: Ethical Statement · p. 8

A Agent Prompt Details

• The planning prompt asks the agent to decompose a task into JSON-formatted steps based on the existing screen image.Examples include opening applications, searching, browsing, and answering questions.Source: A Agent Prompt Details · p. 10
• The acting prompt exposes mouse and keyboard operations, including clicks, double-clicks, scrolling, dragging, key presses, text entry, and waiting.Mouse coordinates are specified relative to the screen’s top-left corner.Source: A Agent Prompt Details · pp. 10–11
• The reflecting prompt asks whether the current image satisfies the subtask and returns success, retry, or plan-reformulation decisions in JSON.Retry and reformulation outputs may include advice for improving execution or revising the plan.Source: A Agent Prompt Details · pp. 11–12

B.1 COCO & Widget Captions Dataset

• COCO and Widget Captions object detections become mouse-click or drag tasks using object centers and bounding-box corners.The conversion preserves each image’s object caption and bounding box as action supervision.Source: §B.1 · p. 12

B.2 Mind2Web Dataset

• Mind2Web sessions are transformed into ScreenAgent-compatible CLICK, SELECT, and TYPE actions using screenshots before and after each action.The source sessions contain multiple steps for completing tasks on one website.Source: §B.2 · p. 13
• A planning template is added for web scenarios, allowing agents to schedule multiple actions when they can be performed on the same page.This augments the model’s planning ability without requiring a page transition between those actions.Source: §B.2 · p. 13

C Details of ScreenAgent Dataset

• Mouse actions constitute the majority of the dataset’s five action types, excluding planning and evaluation actions.This distribution is described as aligning with typical human computer interaction.Source: C Details of ScreenAgent Dataset · p. 14
• ScreenAgent task prompts, action counts, and token usage are summarized through Fig. 9 and Table 5, with Chinese prompts consuming more tokens than English prompts.The dataset also averages 1.5 control actions per screen interaction.Source: C Details of ScreenAgent Dataset · p. 14, Figure 9 · p. 14, Table 5 · p. 14
• About 60% of tasks require 3–5 planning steps, while the most complex tasks require up to 13 steps.The average plan length for these tasks is 4 steps.Source: C Details of ScreenAgent Dataset · p. 14

D CC-Score

• CC-Score compares label and predicted action sequences using action-level similarity scores organized in an n×m matrix.The metric supports action-specific similarity, including action type and mouse-coordinate consistency for mouse actions, and text or key similarity for text and keyboard actions.Source: D CC-Score · p. 14
• The metric selects the highest-scoring order-preserving alignment, allowing unmatched label or predicted actions through null values.The final score is computed from the similarity scores of the selected alignment.Source: D CC-Score · p. 14

E More Generated Samples of ScreenAgent

• Additional generated samples illustrate ScreenAgent’s performance in accomplishing tasks.These examples are presented as concrete demonstrations of the system’s behavior.Source: E More Generated Samples of ScreenAgent · p. 14
• The examples are intended to show how ScreenAgent progresses through planning, action, and reflection.The passage frames the samples around the complete three-phase process.Source: E More Generated Samples of ScreenAgent · p. 14
• The sample section connects task completion with the agent’s three-phase control process.Its purpose is demonstration rather than a separate quantitative evaluation.Source: E More Generated Samples of ScreenAgent · p. 14

F ScreenAgent Training Configurations

• The ScreenAgent model’s training hyper-parameters are documented in Table 6.The supplied passage identifies the table as the source of the training configuration.Source: F ScreenAgent Training Configurations · p. 14, Table 6 · p. 14
• The supplementary examples include a multi-step task for watching a seated-rowing tutorial, beginning with opening a browser and entering a video-site URL.The illustrated plan continues with searching for the tutorial and watching the appropriate result.Source: F ScreenAgent Training Configurations · p. 15, Figure 10 · p. 15
• Additional figures show a complete test-set task, presentation-slide deletion, planning examples, natural-image tests from COCO, and failure cases with golden clickable areas marked by red boxes.These figures broaden the supplementary examples beyond the seated-rowing task.Source: Figure 11 · p. 16, Figure 12 · p. 17, Figure 13 · p. 18