PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoGUI: A Benchmark for GUI Automation from Instructional Videos

Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen WU, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, Mike Zheng Shou

arXiv:2406.10227v1cs.CVcs.AI

TL;DR

Many GUI benchmarks focus on simple, text-described tasks, while advanced tasks often depend on visual cues. VideoGUI evaluates these tasks through hierarchical planning and action tests; results show planning is a major bottleneck, especially from visual previews.

  • Problem

    Many existing GUI benchmarks restrict evaluation to simple tasks describable with a single text instruction, whereas advanced tasks rely more on visual signals.

    Source: §1 · p. 1

  • Method

    VideoGUI derives complex tasks from instructional videos and evaluates assistants on hierarchical planning and atomic action execution.

    Source: §1 · p. 2

  • Results

    GPT-4o scores 39.4 overall, yet high-level visual planning is the lowest-scoring track and GPT-4o fails to complete a single full task.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    VideoGUI provides hierarchical annotations and metrics intended to reveal model limitations and areas for improvement in visual-oriented GUI automation.

    Source: §5 · p. 8

Abstract

from arXiv · show

Graphical User Interface (GUI) automation holds significant promise for enhancing human productivity by assisting with computer tasks. Existing task formulations primarily focus on simple tasks that can be specified by a single, language-only instruction, such as "Insert a new slide." In this work, we introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI assistants on visual-centric GUI tasks. Sourced from high-quality web instructional videos, our benchmark focuses on tasks involving professional and novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex activities (e.g., video editing). VideoGUI evaluates GUI assistants through a hierarchical process, allowing for identification of the specific levels at which they may fail: (i) high-level planning: reconstruct procedural subtasks from visual conditions without language descriptions; (ii) middle-level planning: generate sequences of precise action narrations based on visual state (i.e., screenshot) and goals; (iii) atomic action execution: perform specific actions such as accurately clicking designated elements. For each level, we design evaluation metrics across individual dimensions to provide clear signals, such as individual performance in clicking, dragging, typing, and scrolling for atomic action execution. Our evaluation on VideoGUI reveals that even the SoTA large multimodal model GPT4o performs poorly on visual-centric GUI tasks, especially for high-level planning.

1 Introduction

VideoGUI targets complex GUI tasks that depend on visual cues and are poorly represented by benchmarks focused on single text instructions. It combines instructional-video-derived tasks with hierarchical annotations and evaluation, finding that leading multimodal models struggle especially with visual planning.

1 Introduction

  • Existing GUI benchmarks often limit evaluation to simpler tasks expressible in one text instruction, unlike advanced tasks that rely more on visual signals.Examples include creating special animation effects rather than inserting a slide.Source: §1 · p. 1
  • VideoGUI derives complex GUI tasks from high-quality web instructional videos and covers 11 applications, 86 full tasks, 463 subtasks, and 2.7K manual action annotations.The full tasks average 22.7 actions each.Source: §1 · p. 2, Figure 1 · p. 2
  • Its evaluation separates high-level procedural reconstruction from visual cues, middle-level action narration from visual state and text, and atomic action execution.The hierarchy supports identifying the level at which an assistant falls short.Source: §1 · p. 2
  • Even GPT-4o fails to complete a single full task, with planning—not action execution—the reported bottleneck.Planning from textual queries is much easier than planning from visual previews for almost all evaluated models.Source: §1 · p. 2

2 Related Works

Prior GUI work spans web, mobile, desktop, and multimodal agents, but VideoGUI centers complex tasks that require following instructional videos and attending to visual goals and state transitions.

Benchmarks GUI Tasks

  • Existing GUI benchmarks include web browsing, mobile navigation, desktop grounding, and executable simulated environments, while AssistGUI uses instructional-video subtitles and metadata.AssistGUI evaluates outcomes by task success or failure.Source: §2 · pp. 2–3
  • VideoGUI compares its benchmark with existing GUI datasets by contrasting instructional-video demonstrations, complex full tasks and subtasks, and hierarchical evaluation.It includes 86 full tasks averaging 22.7 actions and 463 subtasks.Source: Table 1 · p. 3
  • VideoGUI targets complex, long-procedure tasks and evaluates high-level procedures, mid-level action decomposition, and atomic execution with emphasis on visual perception.Its focus is identifying visual goals and transitions between states.Source: §2 · p. 3

Multi-Modal Agents

  • Training-free GUI agents often translate interfaces into language or HTML before LLM reasoning, while vision-language-action models are pretrained on GUI image-text data.The former can incur closed-source LLM costs and limit visual perception; when to use different agents or tools remains unclear.Source: §2 · p. 3

3 VideoGUI Benchmarks

VideoGUI is built from instructional-video tasks across diverse software and pairs human demonstrations with annotations. Its evaluation decomposes GUI work into planning and execution stages, using metrics tailored to each stage.

3.1 Data Construction

  • The dataset covers media creation, media editing, and media browsing software, with instructional videos selected for practical and novel uses.Participants reproduce the demonstrated effects in a simulator that records cursor and keyboard activity.Source: §3.1 · p. 3
  • Participants describe the overall goal, divide actions into subtasks, identify active UI elements, retain project materials and visual outcomes, and cross-validate annotations.The simulator records interactions, but annotators identify active elements because the simulator does not automatically record them.Source: §3.1 · pp. 3–4
  • VideoGUI contains 178 tasks across 11 applications, including 86 full tasks and 92 simple tasks; the full tasks comprise 371 subtasks, for 463 subtasks total.The dataset includes 2,712 atomic manual actions, with LeftClick most common at 66.2% and RightClick and Scroll each approximately 2%.Source: §3.1 · p. 4

3.2 Evaluation and Metrics

  • VideoGUI decomposes tasks into high-level milestones, middle-level action narrations, and atomic actions such as clicking and typing.This hierarchy addresses the limits of a single success-rate measure for complex tasks, including tasks with over 100 actions.Source: §3.2 · p. 4
  • High-level planning tests milestone reconstruction from visual outcomes, explicit text objectives, or both, and scores generated procedures against ground truth on a 0–5 scale.An LLM critic evaluates key elements and operations for open-ended responses with multiple valid approaches.Source: §3.2 · pp. 4–5
  • Middle-level planning generates precise action narrations from a milestone and visual observation, with text-only and initial-to-end visual-transition modes also evaluated.Responses are scored by an LLM critic.Source: §3.2 · p. 5
  • Atomic execution evaluates clicking, dragging, scrolling, and typing or pressing through measures suited to each action type.Click and drag use location distance and recall; scrolling is a multiple-choice decision, while typing and key presses are checked in a sandbox using precision and recall.Source: §3.2 · pp. 5–6
  • For clicking, normalized distance measures location error and Recall@d measures whether the prediction falls within a threshold empirically set to 100.Drag scoring jointly measures start and end locations, and its recall requires both predictions to fall within the threshold.Source: §3.2 · p. 5

4 Experiments

VideoGUI evaluates baseline models across planning and atomic actions, with GPT-4o scoring highest overall while visual planning and longer procedures remain challenging. Results vary across software and reveal concrete planning and execution errors.

4.1 Baseline settings

  • The evaluation includes multimodal models and text-only LLMs, with supported instruction modalities summarized for each baseline.The models include GPT-4-Turbo, GPT-4o, Claude-3-Opus, Gemini-Pro-V, Qwen-VL-Max, CogAgent, GPT-3.5-Turbo, LLama3-70B, and Mixtral-8x22B.Source: §4.1 · p. 6

4.2 Main Results on VideoGUI

  • 39.4 is GPT-4o’s highest overall VideoGUI score, and it excels across all three tracks.Table 3 reports high-level planning, middle-level planning, four atomic-action categories, and an overall score.Source: §4.2 · pp. 6–7, Table 3 · p. 7
  • For high-level planning, vision-only scores 0.49 versus 2.37 for textual queries; vision plus text yields no significant gain over text alone.GPT-4o shows the strongest visual reasoning, while most models perform similarly with textual-only inputs.Source: §4.2 · p. 7, Table 4 · p. 7
  • Clicking and dragging remain difficult to localize, while OCR improves click and drag performance; keyboard activity generally scores well.GPT-3.5 with OCR reaches 48.7 recall for clicks, and GPT-4o with OCR reaches 11.3 recall for dragging.Source: §4.2 · p. 7, Table 5 · p. 8

4.3 Performance by Task Difficulty.

  • PowerPoint yields the strongest software-specific performance, while Photoshop and professional or newer applications remain challenging.Runway and Stable Diffusion are described as challenging because they are relatively new to the evaluated models.Source: §4.3 · p. 8
  • Middle-level planning scores tend to decrease as the number of actions per task increases.The figure plots middle-level visual-plus-text planning scores by action count.Source: §4.3 · p. 8

4.4 Qualitative Results

  • In the qualitative example, Gemini-Pro-V adds unnecessary styling and formatting operations that produce positioning and scaling discrepancies.GPT-4o identifies the Morph transition, while GPT-4V’s drag receives zero recall when its pixel distance exceeds the threshold.Source: §4.4 · p. 8

5 Conclusion

VideoGUI is a multimodal benchmark for advanced GUI tasks drawn from instructional videos and aimed at professional and novel software. Its long procedures, hierarchical annotations, and evaluation metrics expose challenges in visual-oriented GUI automation.

5 Conclusion

  • VideoGUI combines long procedural tasks, hierarchical manual annotations, and evaluation metrics to identify limitations in advanced GUI automation.The benchmark targets professional and novel software and is sourced from high-quality instructional videos.Source: §5 · p. 8
  • For the Shiba Inu slide example, the human reference includes a Morph transition, while GPT-4o instead specifies a Fly-in animation.The human reference also calls for rotating the dog and repositioning the title on a duplicated slide.Source: §5 · p. 9
  • Gemini-Pro-V’s middle-level sequence adds 3-D model styling beyond the reference’s insertion, positioning, and resizing operations.Its listed steps include opening the formatting controls and selecting a model style.Source: §5 · p. 9

A.1 Data Collection Settings

The data-collection setup records demonstrations, screenshots, and detailed user activity, then uses manual annotation to name interface elements. The appendix also lists baseline model identifiers and providers.

A.1 Data Collection Settings

  • OBS Studio records demonstration videos and screenshots without capturing the cursor, while pynput records click locations, typed content, and scroll distance.Cursor-free screenshots can be used without revealing target coordinates.Source: §A.1 · p. 10
  • Annotators review key recording screenshots with active regions highlighted and enter the corresponding interface-element name.An example label is “Drop-down menu of font color.”Source: §A.1 · p. 10, Figure 7 · p. 10

A.2 Baseline Details

  • The baseline details list model providers and version identifiers, including DeepInfra’s Llama-3 and OpenAI’s GPT-4o.The list also includes Mixtral, GPT-3.5-Turbo, CogAgent, Qwen-VL-Max, Claude-3-Opus, Gemini-Pro-V, and GPT-4-Turbo.Source: §A.2 · p. 10

A.3 Evaluation Settings

VideoGUI scores click and drag precision, keyboard actions, and scrolling with action-specific measures and checks.

Click

  • Bounding-box clicks average ground-truth distance to all four corners, while coordinate predictions use point distance normalized by the farthest screenshot vertex.The bounding-box measure accounts for box area rather than relying only on its center.Source: §A.3 · pp. 10–11, Figure 8 · p. 11

Drag

  • Drag scores average click distances at the start and end points, with endpoint recall requiring both points to meet the threshold.Each endpoint’s distance is normalized by its farthest-vertex distance.Source: §A.3 · p. 11

Type / Press

  • For typing and key presses, models generate control code that is checked for correct, efficient keyboard output and unnecessary actions.Redundant key presses, such as extra Ctrl presses, are to be avoided.Source: §A.3 · p. 11

Scroll

  • Scroll QA pairs expect a scroll direction before the target becomes visible and “no” after it appears.The answer options are randomly shuffled.Source: §A.3 · pp. 11–12, Figure 10 · p. 12

B Benchmark Statistics

The section reports VideoGUI’s software distribution, screenshot resolutions, manual recording time, and frequent GUI terms.

Software distributions

Manual Recording Cost

World Cloud

C Simulator Experiments

The simulator experiments test a GPT-4o agent on PowerPoint tasks and show that planning assistance improves execution, although full-task completion remains difficult.

Real-world Simulator

Full-task execution

  • All GPT-4o agent variants had 0 success rate on full-task execution, while injected ground-truth planning significantly improved outcomes by human-ranked similarity.The human ranking compares final results with the ground truth; the table covers PowerPoint full tasks.Source: C Simulator Experiments · p. 15, Table 7 · p. 15

Subtask competitions

  • On subtasks, GPT-4o achieved 20.0% success, manual plans increased success by 30%, and the agent used 2.1 more actions than manual demonstrations.The action-count redundancy was exacerbated in complex tasks.Source: C Simulator Experiments · p. 15, Table 8 · p. 15

Final outcomes

  • With ground-truth planning, the agent inserted a 3D model but did not match its background color; another example shows accurate typing of “98%” after plan guidance.Source: Figure 15 · p. 16, Figure 17 · p. 17

D Dataset Examples

The dataset examples are visual-preview samples, mainly focused on visual creation or editing, across presentation, video, and image software.

Data samples

Video and animation examples

Image editing examples

Related papers

Loading 2406.10227v1…