GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding

Research gap

• Earlier GUI datasets and benchmarks focus mainly on static interfaces and Web or mobile settings, limiting assessment of dynamic tasks and cross-scenario generalization.The introduction notes that unexpected elements such as pop-up advertisements can disrupt agent task performance.Source: §1 · p. 2

Dataset contribution

• GUI-WORLD contains 12,379 GUI videos spanning websites, desktop and mobile apps, multi-window interactions, and XR environments.The dataset is built from screen recordings and YouTube instructional videos using Human-MLLM collaboration to generate captions, queries, and multi-round conversations.Source: §1 · p. 2

Benchmark findings

• Most MLLMs struggle with GUI-WORLD, indicating limited dynamic GUI understanding across evaluations of nine MLLMs, five keyframe-selection strategies, and six scenarios.The benchmark includes models such as GPT-4o and Gemini-1.5-Pro and aims to assess GUI-oriented understanding capabilities.Source: §1 · p. 2

GUI-VID and analysis

• GUI-VID improves benchmark performance and reaches results comparable to top-performing models, while the paper also examines textual information, keyframe count, image resolution, and vision perception.The authors describe it as an exploratory GUI-oriented video LLM.Source: §1 · pp. 2–3

2.1 OVERVIEW

• GUI-WORLD covers six GUI scenarios, including desktop and mobile platforms, websites, software, and XR, and is divided into 10,702 training and 1,677 test samples.The dataset includes human-annotated keyframes, detailed captions, and diverse QA produced by its curation framework.Source: §2.1 · pp. 3–4

2.2 GUI VIDEO COLLECTION AND KEYFRAME ANNOTATION PROCESS

• The collection pipeline combines student-worker screen recordings of software tasks with YouTube tutorials, which are segmented into clips representing distinct operation sequences.Tutorial videos supplement worker recordings because those recordings can miss rare functions and take substantial time to produce.Source: §2.2 · p. 4
• Because GUI changes can be minimal, the authors manually extract keyframes and annotate operations, purposes, software or websites, mouse actions, and keyboard inputs.The paper says existing extraction algorithms can underperform on GUI videos, for example when frames differ only by slight cursor movement.Source: §2.2 · pp. 4–5

2.3 GUI TASKS GENERATION FROM HUMAN-MLLM COLLABORATION

• A Human-MLLM pipeline supplies GPT-4V with a prompt, descriptions, key information, and human-annotated keyframes to generate GUI captions and question-answer tasks.The task set includes detailed and summarized captions, questions about static GUI content, and questions about dynamic and sequential content.Source: §2.3 · pp. 5–6
• For dynamic and sequential content, generated questions address changing interface elements and ask about order, temporal relations, and anticipated future states.Human annotators review the video and generated QA pairs to correct inaccuracies and hallucinations and supplement questions and answers.Source: §2.3 · p. 6

3.1 EXPERIMENTAL SETUPS

• The evaluation uses five vision LLMs and four video LLMs, LLM-as-a-Judge scores for free-form responses, accuracy for multiple-choice questions, and varied keyframe and caption inputs.Keyframe settings include human selection, linspace, programmatic Katna, and model-based UVD; non-human settings use 10 frames, while the human setting averages 6.719 frames.Source: §3.1 · p. 6

3.2 EMPIRICAL RESULTS

• GPT-4o averaged 84.8% on multiple-choice queries and 3.573 on free-form queries, outperforming open-source video LLMs in zero-shot GUI settings.The section also reports strong Gemini captioning scores in software and iOS environments, but does not provide a direct comparison for those scores.Source: §3.2 · p. 7
• Dynamic GUI content and conversation challenge MLLMs; GPT-4V and GPT-4o struggle with whole-video descriptions and static content, while GUI-VID is stronger on dynamic tasks.Minor GUI variations can change semantic meaning; GUI-VID nevertheless falls short on captioning and static content, linked to deficiencies in backbone pretraining that instruction tuning did not fully address.Source: §3.2 · pp. 7–8
• Detailed textual information slightly outperforms vision-only input for some tasks, but GPT-4V still falls short on static and dynamic content retrieval.The findings highlight visual perception's role in GUI environments, where minor visual changes can affect outcomes.Source: §3.2 · p. 8
• UVD+VIP keyframe selection gives GPT-4o optimal results, and model-based selectors perform competitively with human selection across basic and complex tasks.Embodied-AI selectors capture semantic transitions that linspace and program-based methods miss.Source: §3.2 · p. 8, Table 6 · p. 8

4.1 METHOD: PROGRESSIVE ENHANCEMENT

• The first stage aligns GUI videos with a pre-trained vision encoder and base LLM using descriptions and captions; the second trains on sequence prediction, conversation, retrieval, and reasoning tasks.The stages progress from basic GUI concepts toward more complex interactions.Source: §4.1 · pp. 8–9
• The VideoChat2-based architecture encodes images and video with UMT-L, compresses visual tokens with an instruction-enhanced QFormer, and fine-tunes the components with LoRA and a vision-grounded text-generation loss.The stated supervised fine-tuning objective optimizes output likelihood given input x and output y.Source: §4.1 · pp. 8–9, Figure 7 · p. 9

4.2 EXPERIMENTS

• GUI-VID improves over its baseline by an average of 30% across tasks and scenarios, surpasses Qwen-VL-Max, and matches GPT-4V and Gemini-Pro on captioning and dynamic sequence-reasoning tasks.The two-stage fine-tuning improves performance in all GUI scenarios.Source: §4.2 · p. 9, Figure 8 · p. 9
• Adding GUI image-text captioning data improves preliminary GUI-element understanding over video-only training, while more keyframes correlate with gains, especially for multi-window and software scenarios.The experiments compare video-only with video-and-image training and vary keyframes between 8 and 16.Source: §4.2 · p. 9, Table 8 · p. 10
• GUI-VID still fails on some GUI operating tasks via code generation, which the authors link to weak base-LLM performance and challenges in code-generation instruction fine-tuning.The section describes exploratory experiments on operational-code generation and chat assistance.Source: §4.2 · pp. 9–10

Related work

• MLLMs support GUI applications such as web and mobile agents, with research also exploring computer-game control and operating-system copilots.The broader field also applies MLLMs to visual reasoning, medical-image interpretation, and embodied agents.Source: §5 · p. 10
• GUI datasets and benchmarks build on mobile resources such as Rico and AitW, with research extending across web and desktop environments.Mind2Web includes more than 2,000 tasks from 137 websites across 31 domains.Source: §5 · p. 10
• GUI research has moved from single-image understanding toward sequential operations and multi-hop scenarios.Source: §5 · p. 10

6 CONCLUSION

• GUI-WORLD covers six scenarios and various tasks, addressing a gap in comprehensive evaluation of graphic-based understanding.The dataset is designed to benchmark and enhance understanding of virtual interfaces, especially sequential and dynamic tasks.Source: §6 · p. 10
• The paper benchmarks leading MLLMs and introduces GUI-VID, a video-LLM assistant fine-tuned on GUI-WORLD, reporting results comparable to top-performing models.The authors say the work provides insights for future research in dynamic GUI content understanding.Source: §6 · p. 10

Appendix

• The appendix summary table organizes evaluation results by task and scenario, including capabilities such as captioning, complex QA, conversation, and application domains such as XR and iOS.Source: Table 9 · p. 19

A.1 SIX MAIN GUI CATEGORIES

• The six categories are Android, software, websites, iOS, multi-window interactions, and XR.Source: §A.1 · p. 19
• Multi-window interactions are common in desktop use, yet the authors report no dataset specifically catering to them.Source: §A.1 · p. 19
• XR interfaces take an immersive 3D form, and the authors report no datasets specifically addressing GUI in XR.Source: §A.1 · p. 19

A.2 SELECTED WEBSITE/SOFTWARE

• Website and software selections span social media, productivity, shopping, and education to capture varied user interactions and GUI complexity.Source: §A.2 · p. 21

• Twenty-four annotators—16 authors and 8 volunteers—received tutorials and criteria before independently annotating the data.Source: §A.3 · p. 22
• Self-recorded footage and YouTube tutorials are segmented into goal-focused clips containing multiple actions, then processed into annotated keyframes.Source: §A.3 · p. 22
• Annotators select keyframes using video content and contemporaneous mouse and keyboard actions, label intervening sub-operations, and summarize each video’s goal.Source: §A.3 · p. 22
• Human review of 1,000 GPT-4V descriptions and captions yielded a 98% satisfaction rate.Source: §A.3 · pp. 22–23

• Questions for sequential and prediction tasks are slightly longer than other types, while golden answers for static tasks tend to be longer.Source: B DATASET ANALYSIS · p. 23, Figure 13 · p. 24, Figure 14 · p. 24
• Question-answer lengths are similarly distributed across GUI scenarios, with Android questions slightly shorter and XR answers longer.Source: B DATASET ANALYSIS · p. 24, Figure 13 · p. 24, Figure 14 · p. 24

C.1 FINE-TUNE DATASET CONSTRUCTION

• GUI-VID uses either GUI-WORLD video-text pairs alone or video-text pairs combined with image-text pairs to align the visual encoder with GUI knowledge.Source: §C.1 · pp. 24–25

C.2 HYPERPARAMETER SETTINGS

• The reported benchmark setup covers seven MLLMs, while the experiments also specify inference, judge, and fine-tuning hyperparameters.Source: §C.2 · pp. 25–26

C.3 EVALUATION.

• In a 1,000-sample comparison across six free-form question types, GPT-4 aligns better with human scoring than other tested judges, at approximately 10 times their cost.Source: Table 15 · p. 26, §C.3 · p. 27

D ADDITIONAL EXPERIMENTS RESULTS

• Removing MetaGUI and OmniAct improves MCQA and XR capabilities but reduces performance on general free-form questions relative to models trained with those datasets.The authors report that GUI-WORLD-only models remain highly competitive.Source: D ADDITIONAL EXPERIMENTS RESULTS · p. 27
• Model-based keyframe identification performs competitively with human selection, and GPT-4o achieves its best reported results with UVD+VIP.The comparison covers basic captioning and complex static and dynamic analysis tasks.Source: D ADDITIONAL EXPERIMENTS RESULTS · p. 27
• GUI-understanding results generally match agent-benchmark performance, and human annotators favor GUI-understanding-trained models as GUI assistants.The human-preference study used 180 videos, with 30 from each scenario, and five annotators.Source: D ADDITIONAL EXPERIMENTS RESULTS · pp. 27–28
• GUI-VID outperforms Video-LLaVA and LLaVA-Next-Video-7B-DPO on most tasks, except conversation, where DPO may contribute to their advantage.The authors identify DPO training as a possible explanation for the conversational exception.Source: D ADDITIONAL EXPERIMENTS RESULTS · p. 28
• On MVBench, GUI-VID lags its original model but significantly outperforms two other baseline models.The authors note that the training used neither the strongest baseline nor optimal methods, which may have contributed to catastrophic forgetting.Source: D ADDITIONAL EXPERIMENTS RESULTS · p. 28, Table 21 · p. 29
• Fine-tuning on GUI-WORLD improves VideoChat2 and GUI-VID performance on Mind2Web-Multimodal operational benchmarks.The evaluation included zero-shot and Mind2Web-fine-tuned settings for both models.Source: D ADDITIONAL EXPERIMENTS RESULTS · p. 28

E PROMPTS

• The appendix documents human-annotation guidelines, task-generation and benchmarking prompts, and separate LLM-as-a-Judge prompts for free-form, conversational, and multiple-choice QA.The task-generation prompt explicitly requests sequential questions requiring understanding of GUI changes across video frames.Source: E PROMPTS · pp. 29, 36, 40

F CASE STUDY

• Case studies present human-annotated frames and associated tasks in two parts for each of six scenarios: Android, iOS, multi-window interaction, website, XR, and software.Source: F CASE STUDY · p. 41