Source-linked AI summary
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, Ruslan Salakhutdinov
TL;DR
Computer-task agents must interpret natural-language instructions and screenshots while executing precise action sequences across web and desktop applications. OmniACT provides a large benchmark for this setting and finds that current models, including GPT-4, remain challenged, motivating multimodal agent research.
Problem
Existing autonomous-agent benchmarks and methods focus mainly on web or Android environments, leaving desktop and cross-application computer tasks insufficiently assessed.
Method
OmniACT pairs screenshots and natural-language tasks with executable PyAutoGUI scripts across web and desktop applications, and benchmarks language and multimodal agents.
Results
GPT-4 achieves the strongest reported performance, while OmniACT remains challenging for current language and multimodal models.
Takeaways & Limitations
The benchmark supports future development of multimodal models that integrate language understanding with visual understanding of computer screens.
Takeaways & Limitations
The dataset is English-only and human-curated content may introduce temporal biases, while reliance on closed models creates cost and time constraints.
Abstract
from arXiv · showhide
For decades, human-computer interaction has fundamentally been manual. Even today, almost all productive work done on the computer necessitates human input at every step. Autonomous virtual agents represent an exciting step in automating many of these menial tasks. Virtual agents would empower users with limited technical proficiency to harness the full possibilities of computer systems. They could also enable the efficient streamlining of numerous computer tasks, ranging from calendar management to complex travel bookings, with minimal human intervention. In this paper, we introduce OmniACT, the first-of-a-kind dataset and benchmark for assessing an agent's capability to generate executable programs to accomplish computer tasks. Our scope extends beyond traditional web automation, covering a diverse range of desktop applications. The dataset consists of fundamental tasks such as "Play the next song", as well as longer horizon tasks such as "Send an email to John Doe mentioning the time and place to meet". Specifically, given a pair of screen image and a visually-grounded natural language task, the goal is to generate a script capable of fully executing the task. We run several strong baseline language model agents on our benchmark. The strongest baseline, GPT-4, performs the best on our benchmark However, its performance level still reaches only 15% of the human proficiency in generating executable scripts capable of completing the task, demonstrating the challenge of our task for conventional web agents. Our benchmark provides a platform to measure and evaluate the progress of language model agents in automating computer tasks and motivates future work towards building multimodal models that bridge large language models and the visual grounding of computer screens.
1 Introduction
OmniACT addresses the limitations of existing computer agents by benchmarking visually grounded task execution across web and desktop applications. It introduces a large dataset, a screen-understanding module, and evaluations showing that current models remain far below human performance.
- Existing agents often target web applications, struggle with complex HTML, and cannot reliably operate native desktop or cross-application tasks.
- OmniACT contains over 9.8K image-instruction pairs spanning operating systems and the web, enabling evaluation of visually grounded computer-task automation.
- The benchmark evaluates language and multimodal agents, including GPT-4, GPT-4 Vision, LLaMA, Vicuna, and Gemini-Pro.
- The paper contributes a desktop-and-web dataset, custom computer-task metrics, the DetACT screen-representation module, and a benchmark of state-of-the-art models.
- OmniACT extends prior UI-understanding and web-agent benchmarks toward autonomous agents that combine language planning with visual screen understanding.
Reverse Mapping and Filtering
The data collection pipeline converts diverse application screens into executable, visually grounded tasks through annotation, task generation, coordinate mapping, and filtering.
- The pipeline selects over 60 applications and websites, segments screens with human-annotated bounding boxes, and labels the boxes by functionality.
- Student volunteers generate tasks from screen images, after which textual labels are reverse-mapped to coordinates and scripts are filtered by execution and syntax.
3 OmniACT
OmniACT defines computer-task automation as generating PyAutoGUI action sequences from a screen and natural-language instruction, using broad desktop and web coverage with visually grounded annotations. Its collection pipeline produces executable gold-standard scripts across diverse applications, platforms, and task intents.
- 3 OmniACT: OmniACT measures autonomous agents on both web and desktop applications using natural-language tasks, UI screenshots, and successful PyAutoGUI scripts.
- 3.1 Task Formulation: Given screen S and task T, the objective is to learn a transition function f: T × S → A that outputs executable actions A.
- 3.1 Task Formulation: The action space includes clicks, drags, scrolling, writing, and other PyAutoGUI operations, with mouse actions requiring screen coordinates.
- Application/Website Selection: The dataset contains 9802 data points across desktop and web applications, with desktop applications emphasized because they lack DOM hierarchies and require visual cues.
- Application/Website Selection: Tasks cover desktop applications, 27 web applications, and six intent categories: Shopping, Entertainment, Service, Government, Travel, and Health.
- Task Creation: Human annotation identifies UI regions, labels their functionality, and supports visually grounded tasks designed to execute within a single screen.
- Task Creation: DetACT combines OCR, SAM, icon and color matching, and an LLM filter to select task-relevant UI elements from screen images.
- Task Creation: After filtering and manual review, the dataset contains 9802 human-annotated gold-label points across more than 200 desktop and web screens, split 7:1:2.
4 Evaluation Metrics
OmniACT evaluates generated scripts with metrics that separately assess exact action-sequence matching and execution quality, including spatially aware penalties for incorrect clicks, keys, and typed text.
- 4.1 Sequence Score: Sequence score measures whether the predicted action sequence exactly matches the gold sequence.The score increases with sequence length when all actions match and is zero otherwise; β1 is 0.1 and β2 is 1.
- 4.2 Action Score: Click penalties apply when predicted coordinates fall outside a target UI element’s bounding box and vary with Euclidean distance and box size.L2 is the smallest distance from the prediction to the bounding box, while μ is set to the inverse diagonal length.
- 4.2 Action Score: Key penalties compare the target and predicted key sets for press and hotkey actions.The metric checks whether the corresponding key sets are identical.
- 4.2 Action Score: Write penalties use BLEU to penalize differences between the target sentence and the sentence predicted for a write action.The penalty is applied when the sequence score is positive and compares the gold and predicted strings.
- 4.2 Action Score: The action score weights penalties by α_i, defined as the sequence score divided by the sequence length.The mean action score aggregates the weighted penalties across examples.
5 DetACT: DETecting ACTions from UI
DetACT converts UI screenshots into structured textual and visual signals for downstream action generation, then filters those elements according to the task.
- DetACT overview: DetACT converts UI images into structured code and text outputs because HTML-based descriptions may omit important application components.Its three modules are text, icon, and color processing.
- Text Extraction: The text module extracts UI text and records the locations of text-based elements.It uses EasyOCR and segments screenshot regions with SAM before filtering non-textual regions for later modules.
- Baseline Model Architecture: The baseline architecture feeds the filtered UI-element list and task into a prompt, with the image additionally supplied to multimodal LLMs.The LLM then generates the automation script.
- Icon Module: The icon module matches segmented regions against a template pack of 1600 labeled icons.Reference icons and regions of interest are resized and converted to grayscale for similarity comparison.
- Task-conditioned filtering: DetACT filters extracted UI elements by prompting GPT-4 with the task description and the list of candidate elements.The selected elements are intended to be relevant to the requested action.
6 Baselines
The baselines include few-shot and fine-tuned generative language models alongside multimodal models, using DetACT-derived screen descriptions and, for multimodal systems, the screen image.
- Few-shot Generative LLM: Prompt-based baselines include LLaMA, Vicuna-1.5, CodeLLaMA-34B, Palmyra, and GPT-series models.The models receive the prompt structure shown in the baseline architecture.
- Few-shot Generative LLM: Lower-confidence DetACT outputs are removed for LLaMA and CodeLLaMA to reduce prompts to 2000 tokens, while other models use prompts up to 4000 tokens.The authors report poor performance with longer prompts for LLaMA and CodeLLaMA.
- Finetuned Generative LLM: LLaMA-13B and Vicuna-13B are fine-tuned with QLoRA for code generation from DetACT screen descriptions and task instructions.The fine-tuning setup uses rank 64, scaling factor 16, and 300 steps.
- Few-shot Generative Multimodal Models: Few-shot multimodal baselines receive a similar prompt together with the screen image.This setting reflects that most OmniACT tasks are visually grounded.
7 Results and Analysis
GPT-4 performs best among the evaluated baselines, while multimodal inputs improve coordinate prediction and GPT-4 Vision outperforms GPT-4 on action score. Human evaluators generally show high first-attempt proficiency but still encounter grounding and interface-familiarity difficulties.
- Baseline comparison: GPT-4 achieves the strongest baseline performance, with higher sequence scores and lower coordinate-prediction and text-input penalties.The benchmark compares prompt-based LLMs, fine-tuned LLMs, and prompt-based multimodal models using sequence, action, and penalty metrics.
- Fine-tuned models: Fine-tuning improves LLaMA-13B’s sequence score from 4.80 to 8.92 and action score from 1.62 to 2.14.Fine-tuned models also outperform their few-shot prompt-only counterparts on other metrics.
- Fine-tuned models: Text-only prompt and fine-tuned LLMs incur severe mouse penalties, especially for click coordinates, because they rely solely on textual signals.The limitation is tied specifically to coordinate prediction in screen-based tasks.
- Multimodal models: Providing the full screen image significantly improves coordinate prediction in multimodal language models.The image enables fuller use of the screen representation during action generation.
- Multimodal models: GPT-4 Vision outperforms GPT-4 on Action Score and also improves sequence score, combining GPT-4’s reasoning with stronger visual understanding.The authors attribute the improvement to the combination of reasoning and visual understanding capabilities.
- Human performance: Human evaluators generally perform proficiently on first attempts, but difficulties arise from task comprehension, screenshot grounding, and unfamiliar interfaces.These factors explain instances in which users do not successfully complete tasks.
8 Conclusion and Future Work
OmniACT introduces a human-labeled benchmark for autonomous agents across web and desktop applications. GPT-4-based agents achieve an action score of 11.6, while the benchmark remains challenging for current language and multimodal models.
- Conclusion: OmniACT is a 9.8K human-labeled dataset and benchmark spanning autonomous-agent tasks on web and desktop applications.The benchmark is presented as a platform for evaluating current agents and motivating future multimodal research.
- Conclusion: 11.6 is the action score achieved by GPT-4-based agents on OmniACT.The reported score is described as respectable, but the benchmark remains challenging for current state-of-the-art language and multimodal models.
- Future work: OmniACT motivates future work on foundational multimodal models that integrate language with visual understanding of computer screens.The stated direction concerns generalist autonomous agents assisting humans with computer tasks.
9 Limitations
The authors identify limitations involving model hallucinations and data-type bias, closed-model integration costs, and potential English-language and temporal biases in the human-curated dataset.
- Model limitations: GPT-4-class models may be susceptible to hallucinations and bias toward specific data types, limiting broad applicability.This limitation is stated for state-of-the-art models such as GPT-4.
- Deployment limitations: Reliance on closed models such as GPT-4V creates integration challenges because of high costs and time constraints.The constraint concerns practical integration of closed-model systems.
- Dataset limitations: An English-only dataset and human-curated content may introduce language and temporal biases despite efforts toward equal representation and privacy.The authors state that data collection avoided personal information while acknowledging these remaining bias risks.
10 Ethics Statement
The dataset-creation pipeline includes reviews for personally identifiable information and offensive content, uses dummy accounts, and obtains scripts and bounding-box annotations from approved workers.
- Data review: The pipeline reviews each stage for personally identifiable information and offensive content during data collection and LLM use.The authors state that these checks are applied throughout dataset creation.
- Data review: Dummy accounts are created to mimic realistic user content without using real personal accounts.This practice is part of the dataset-creation process.
- Annotation process: Gold-label scripts are collected with help from institution-approved student workers, while bounding-box data is annotated through MTurk workers.The ethics statement identifies separate worker groups for scripts and bounding-box annotations.