GUICourse: From General Vision Language Models to Versatile GUI Agents

1 Introduction

• Current VLMs can lack accurate OCR and grounding as well as knowledge of GUI widget functions and control mechanisms.The paper illustrates these challenges with Qwen-VL-Chat on GUI navigation tasks.Source: §1 · p. 1
• GUICourse includes 67k human-verified single-step and 5,696 human-annotated multi-step navigation instructions, plus GUIEnv and GUIChat data.The introduction describes GUIEnv as 10 million website page-annotation pairs and GUIChat as 44k single-turn QA pairs and 6k multi-turn dialogues.Source: §1 · p. 2
• The proposed pipeline has two learning stages: strengthen fundamental abilities, then acquire GUI-specific knowledge.The datasets provide training data for building GUI agents from VLMs.Source: §1 · p. 2
• Even small-sized visual GUI agents perform effectively on navigation tasks, and OCR and grounding data during pretraining helps those tasks.The introduction also reports that several agents were trained using the proposed pipeline and datasets.Source: §1 · p. 2

2.1 Vision Language Models

• VLMs can recognize sparse characters or dense text with similar fonts, but may struggle with varied fonts and pixel-level text locations in large screenshots.The paper presents GUIEnv as a dataset intended to improve OCR and grounding using high-resolution website screenshots.Source: §2.1 · p. 2

2.2 GUI Agents

• GUI agents vary in perception modality and whether actions require positions; this work targets vision-only inputs and position-based actions.The paper notes that text extraction can be difficult under system permission restrictions, while screenshots are readily available.Source: §2.2 · p. 3

3 GUICourse

• The three GUICourse partitions address OCR and grounding, GUI knowledge, and agent interaction, respectively.Dataset construction combines LLM-based auto-annotation and human annotation.Source: §3 · p. 3

3.1 GUIEnv: A Large Scale Dataset for OCR and Grounding Abilities

• GUIEnv has 10M full-page samples for pretraining and 0.7M region-focused QA samples for SFT, supporting OCR and grounding.The local subset includes text2bbox and bbox2text tasks.Source: §3.1 · p. 3
• GUIEnv-global samples are long full-page descriptions with text, grounding information, and layout sequence; GUIEnv-local samples are region-specific QA.The global subset is for pretraining and the local subset for SFT.Source: §3.1 · p. 3

3.2 GUIAct: A GUI Navigation Dataset with Single-step and Multi-step Instructions

• GUIAct provides 67k single-step and 5,696 multi-step website navigation instructions, alongside smartphone navigation data.Its website subsets use screenshots as inputs and a unified action space that includes position-bearing actions.Source: §3.2 · p. 4
• Human checking raised sampled GUIAct web-single annotation accuracy from 55% to 92%.Annotators revised inaccurate automatic pairs or abandoned items that were difficult to revise.Source: §3.2 · p. 4

3.3 GUIChat: A Text-Rich Multi-Modal QA Dataset for Interaction

• GUIChat contains about 44k single-turn QA pairs and 6k multi-turn dialogues linked to text-rich website screenshots and grounding boxes.The dialogues cover visual-information queries, human-centric issues, world knowledge, and complex reasoning.Source: §3.3 · p. 5
• GUIChat generates question-answer pairs from structured website text and coordinate information extracted from page screenshots and DOM trees.The dataset is designed to mix conversation data into GUI-agent training.Source: §3.3 · p. 5

4 Experiments

• The experiments assess GUICourse-trained agents on test datasets and common GUI tasks, then study training factors and illustrate agent generalization with examples.MiniCPM-V is the basis for the training-process ablations.Source: §4 · pp. 5–6

4.1 Training Settings

• MiniCPM-GUI is trained with GUIEnv-global pre-training data, followed by SFT using GUIEnv-local, GUIAct, and GUIChat data.Qwen-GUI and Fuyu-GUI are trained only with SFT data.Source: §4.1 · p. 6

4.2 Performance on GUI Navigation Tasks

• The experiments evaluate agents on held-out GUIAct samples and on Mind2Web and AITW after fine-tuning agents and baseline VLMs for those tasks.The held-out sets contain about 1.4k web-single, 1k web-multi, and 2k smartphone samples.Source: §4.2 · p. 6
• On the three in-house test datasets, Qwen-GUI (9.6B) reaches mean StepSR 57.2, while MiniCPM-GUI (3.1B) reaches 57.1.The reported mean StepSR rises to 56.1 from 49.0 with high resolution, and GUIChat improves website mean StepSR by 1 point.Source: §4.2 · p. 6, Table 3 · p. 7
• GUICourse agents outperform their baseline VLMs on Mind2Web and AITW, with larger Mind2Web gains for Fuyu-8B and MiniCPM-V than for Qwen-GUI.AITW improvements are relatively smaller, which the authors attribute as likely related to using more data in website scenarios.Source: §4.2 · pp. 6–7, Table 4 · p. 7, Table 5 · p. 7

4.3 Ablation Study

• As GUIEnv data increases, text2bbox, bbox2text, and web-single performance rise; Type EM is unchanged because action-name prediction is position-free.The authors report a large improvement between no GUIEnv data and 2.5M GUIEnv samples.Source: §4.3 · p. 7, Table 6 · p. 8

4.4 Case Study

• In the simulated smartphone example, MiniCPM-GUI swipes to find Settings when absent and taps it when visible, rather than repeating one fixed action sequence.The authors interpret this behavior as distinguishing screenshot changes and recognizing the Settings icon.Source: §4.4 · pp. 7, 9, Figure 4 · p. 8

4.5 Error analysis

• Among 50 reviewed web-multi errors, 13 were reasonable outputs mismatched to the gold answer, 22 were action-type errors, and 15 were action-position errors.The authors also report that complex screenshot content can prevent correct actions.Source: §4.5 · p. 9

5 Conclusion

• GUICourse combines GUIEnv, GUIAct, and GUIChat to target OCR and grounding, GUI knowledge, and interaction ability, respectively.The authors describe a comprehensive pipeline for training visual-based GUI agents from general VLMs.Source: §5 · p. 9
• The authors evaluate trained GUI agents on GUI navigation tasks and try them in simulated Android environments.Source: §5 · p. 9
• Ablation studies examine the influence of OCR and grounding abilities.Source: §5 · p. 9

Limitations

• Pretraining and supervised fine-tuning may not be enough to achieve assistant-level GUI agents.The authors are considering reinforcement learning methods such as RLHF for future work.Source: Limitations · p. 9
• The training data covers website and smartphone environments, but the authors say these scenarios alone are insufficient for general GUI agents.Source: Limitations · p. 9
• The authors plan to build or gather GUI instructions for computer systems and professional software.Source: Limitations · p. 9

A.1 Action Space

• The action space organizes task-oriented GUI actions into seven groups, and excludes actions whose functions can be replaced by others in the same group.The groups cover pointing, input, browsing, logging, submitting, selecting a value, and ending a task.Source: §A.1 · p. 12

A.2 LLM-Auto Annotation

• The authors use LLMs in the data-creation process for GUIAct single- and multi-step navigation and GUIChat.Source: §A.2 · p. 13

A.2.1 GUIAct (single-step website navigation)

• For single-step website navigation, GPT-4V receives an original screenshot and a version with interactive elements indexed and masked to generate instruction-action pairs.The original image provides viewport content, while the revised image provides element IDs.Source: §A.2.1 · p. 13

A.2.2 GUIAct (multi-step website navigation)

• For multi-step website navigation, GPT-3.5 and Claude2 generate English and Chinese questions for matching-language websites, with answers tied to website information.Source: §A.2.2 · p. 13

A.2.3 GUIChat

• GUIChat uses GPT-4 to generate single-turn and multi-turn question-answer pairs, with multi-turn generation conditioning each round on conversation history and allowing up to five rounds.Single-turn generation begins with four question types, followed by answers and grounded answers.Source: §A.2.3 · p. 13

A.3 Crowd-sourced Annotation

• Human annotators help check or annotate GUIAct single- and multi-step website navigation data and GUIChat data.Source: §A.3 · p. 13

A.3.1 GUIAct (single-step website navigation)

• Accuracy rose from 55% to 92% after review of about 78k single-step instances, yielding 67k samples after corrections and removal of problematic items.The review process fixed errors such as incorrect element IDs or action names and deleted more complex problematic samples.Source: §A.3.1 · p. 13

A.3.2 GUIAct (multi-step website navigation)

• Multi-step GUIAct annotation followed a handbook; annotators were paid 5 RMB per item, with approximately five minutes spent annotating each item.The annotation phase lasted four months, from September 2023 to January 2024.Source: §A.3.2 · p. 14

A.3.3 GUIChat

• Human evaluation scores for GUIChat were 0.954 for Helpfulness, 0.814 for Honesty, 0.958 for Harmlessness, and 0.886 for Image-text correlation.The evaluation sampled 500 question-answer pairs; the authors report that most errors stemmed from screenshots of unloaded websites.Source: §A.3.3 · pp. 14–15

• GUIEnv contains 10M pre-training samples and 0.7M SFT samples, with 1.8k GUIEnv-local samples held out for OCR and grounding evaluation.Source: §B.1 · p. 15
• GUIAct includes about 67k web-single instructions, 5,696 human-annotated web-multi instructions, and 9,157 smartphone instructions sampled from AITW.The three partitions contain 67k, 44k, and 67k samples, respectively; about 1.4k, 1k, and 2k are test samples.Source: §B.1 · pp. 15–16
• GUIChat provides 44k single-turn QA pairs and 6k multi-turn dialogues, all used for SFT.A prompt template covers questions about information inquiry on a designated website.Source: §B.1 · p. 16, Figure 9 · p. 16
• Pre-training targets OCR and text grounding, while SFT targets GUI-system knowledge, including element functions and control actions, as well as conversation abilities.Source: §B.1 · p. 16
• MiniCPM-V receives GUIEnv-global mixed into pre-training, then SFT on 10% of GUIEnv-local plus all GUIAct and GUIChat data; Qwen-VL and Fuyu-8B receive SFT only.MiniCPM-V uses a 5e-7 learning rate, versus 1e-5 for Qwen-VL and Fuyu-8B.Source: §B.1 · p. 16
• Training MiniCPM-V uses 128 A100 GPUs for pre-training and 8 for SFT; its reported training time is about two days and eight hours, respectively.Qwen-VL and Fuyu-8B each take about one day for SFT; inference uses one A100 GPU per agent.Source: §B.1 · p. 16

B.2 The Evaluation of Actions.

• For click and hover, IoU scores regions while success compares element IDs exactly; the predicted region is assigned to its nearest candidate element.The figure illustrates the IoU metrics and nearest-element attachment method.Source: §B.2 · p. 16, Figure 10 · p. 17
• Tap success uses a distance threshold of 0.14, with action score 1 − distance below that threshold and zero otherwise.Source: §B.2 · p. 17
• Input and answer use F1 with a 0.5 success threshold; scroll and swipe compare directions, while copy and enter use Type EM.Source: §B.2 · pp. 16–17
• Select averages IoU for its element and F1 for its text, and succeeds only when both parameters meet their criteria.Select_text succeeds when IoU exceeds 0.5.Source: §B.2 · p. 17
• Step Success Rate requires all actions in a step to succeed, after comparing the top-n predictions with golden actions and averaging their action scores.Missing predicted actions receive a score of 0.0.Source: §B.2 · pp. 17–18

B.3 Case Study

• The case study shows a long-chain example of an agent executing instructions in a simulated smartphone environment.Source: §B.3 · p. 18

B.4.1 MiniCPM-GUI with Larger LM

• The larger MiniCPM-GUI performs better on the authors’ test data and outperforms SeeClick on Mind2Web.The agent combines GUIEnv data with its pre-training data and is fine-tuned on GUIAct and GUIChat; the Mind2Web comparison is presented in Table 9.Source: §B.4.1 · p. 18, Table 9 · p. 18

• Adding GUIChat to GUIAct produces similar action-execution results to GUIAct alone on MiniCPM-V and Qwen-VL.The authors interpret this as GUIChat having little impact on GUI navigation ability.Source: §B.4.2 · p. 18

C Data Examples

• Examples cover GUIEnv global and local tasks, GUIAct web-single and web-multi actions, and GUIChat single- and multi-turn QA.The examples show text-and-position prediction or grounding, action labels and screenshots, and QA dialogues.Source: C Data Examples · p. 18, Figure 14 · p. 22, Figure 15 · p. 23

D Ethical Considerations and Societal Impacts

• The authors report fair wages for annotators, no personally identifiable information in their data, and citation of creators for existing assets.They also state that they use open-source datasets or models.Source: D Ethical Considerations and Societal… · p. 18
• Some GUIEnv screenshots come from Cleaned Common Crawl, so the authors cannot guarantee they are free of personally identifying information or offensive content.Source: D Ethical Considerations and Societal… · p. 18