PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

Felix Henry, Xiaochen Lin, Jiangyou Zhu, Yangfan, Bingqian Zhang, Min Chen, Shiyu Huang

arXiv:2605.18758v1cs.HCcs.AI

TL;DR

Existing GUI benchmarks largely miss the transient, step-specific audio and video signals involved in real-world smartphone interaction. OmniGUI evaluates agents with continuous interleaved images, synchronous audio, and temporal video at each action step, finding that the best model reaches 66.4% Exact Match accuracy while performance falls on temporally and auditorily dependent tasks.

  • Problem

    Existing GUI benchmarks largely assess static or passive multimodal inputs, missing sequential action decisions grounded in step-specific audio and video signals.

  • Method

    OmniGUI benchmarks sequential smartphone GUI decisions using continuous interleaved screenshots, synchronous audio, and temporal video at every action step.

  • Results

    66.4% Exact Match step accuracy is achieved by the highest-performing model, while performance degrades on AV-Critical tasks without non-visual modalities and under cross-modal interference.

  • Takeaways & Limitations

    OmniGUI shows that current multimodal agents remain challenged by synchronous temporal and auditory signals despite competency on static visual tasks.

  • Takeaways & Limitations

    The offline, expert-demonstrated step-level protocol does not evaluate recovery from compounding errors during autonomous end-to-end rollouts.

Abstract

from arXiv · show

Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action. To bridge this gap, we introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments. OmniGUI provides continuous, interleaved multimodal inputs comprising static images, synchronous audio, and video clips at every action step. The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications, systematically annotated with objective multimodal dependency levels. Because dedicated omni-modal GUI agent frameworks are currently in their nascent stage, we select foundational omni-modal models capable of natively processing interleaved inputs to serve as agent proxies for our initial baselines. Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals. Furthermore, ablation studies isolate specific operational bottlenecks, notably cross-modal interference when processing task-irrelevant environmental noise. The complete dataset, evaluation pipeline, and baseline prompts are provided in the supplementary material. Project page: https://omni-gui.github.io.

1 Introduction

OmniGUI addresses the limitations of screenshot-centered GUI benchmarks by evaluating smartphone agents on continuous, interleaved image, audio, and video inputs at every action step. Its expert-demonstrated dataset and model baselines show that transient multimodal signals challenge current agents and expose cross-modal operational bottlenecks.

  • Existing GUI benchmarks predominantly use static screenshots, while newer multimodal benchmarks often provide audio or video only as pre-task reference content.Real-world device interaction instead couples multimodal signals to the moment of action.
  • OmniGUI is the first benchmark evaluating GUI agents with continuous, interleaved static images, synchronous audio, and temporal video clips at every action step.Agents predict precise action primitives and parameters from a 13-action space using these synchronous inputs and action history.
  • 709 expert-demonstrated episodes and 2,579 action steps span 29 mobile applications and are annotated with objective multimodal dependency levels.The dataset is organized around five cognitive operational dimensions and three dependency levels: AV-Critical, AV-Supportive, and AV-Present.
  • 66.4% is the highest Exact Match (EM) step accuracy achieved across eight proprietary and open-source models, highlighting difficulty with transient multimodal action execution.Foundational omni-modal models, including Gemini 3.0 Pro and Qwen3-Omni, serve as initial agent proxies because dedicated frameworks remain nascent.
  • Ablations show significant degradation on AV-Critical tasks when non-visual modalities are removed, while static AV-Present tasks remain largely unaffected.The evaluation also identifies cross-modal interference from irrelevant signals and degradation during concurrent dual-audio processing.

2 Related Work

Existing GUI benchmarks largely evaluate agents from static screenshots, omitting the auditory and temporal dynamics of real-world interaction. Recent multimodal benchmarks and omni-modal model evaluations broaden inputs but generally assess pre-task materials or passive understanding rather than synchronous step-level interaction.

  • GUI Agent Benchmarks: Most GUI agent benchmarks use exclusively static screenshots across Android, web browsers, desktop operating systems, and cross-platform element grounding.These benchmarks established a foundation for agentic automation but omit auditory and temporal dynamics ubiquitous in real-world environments.
  • Multimodal GUI Benchmarks: Recent benchmarks incorporate audio transcriptions, video demonstrations, or embedded multimedia content for multimodal interaction analysis.Examples include Multimodal-Mind2Web, GUI-World, VideoGUI, and VideoWebArena.
  • Multimodal GUI Benchmarks: These multimodal GUI benchmarks predominantly treat audio and video as pre-task reference materials rather than synchronous step-level inputs.This limitation distinguishes their modality handling from the step-level synchronous setting targeted by OmniGUI.
  • Omni-Modal Model Evaluation: Foundational omni-modal models now natively process interleaved text, image, audio, and video through proprietary and open-source initiatives.Examples include GPT-4o, the Gemini family, Qwen3-Omni, MiniCPM-o, and VITA.
  • Omni-Modal Model Evaluation: Despite diverse multimodal evaluations, existing benchmarks strictly assess passive perception and understanding using fixed media inputs.Their outputs are textual answers or classification labels rather than actions in an interactive environment.

3 The OmniGUI Benchmark

OmniGUI models smartphone GUI interaction as sequential action prediction from interleaved screenshots, video, audio, and action history. Its benchmark contains 709 episodes across 29 applications, organized by cognitive task dimensions and objectively annotated multimodal dependency levels.

  • Sequential Interaction Formulation: At each step, the agent receives instruction G and multimodal state S_t=(I_t,V_t,A_t,H_t) before predicting executable action a_t.I_t is the current screenshot, V_t records preceding screen dynamics, A_t is synchronous audio, and H_t is the historical action trajectory.
  • Sequential Interaction Formulation: The action space contains 13 operational primitives spanning five categories, with coordinates normalized to the resolution-independent [0, 1000] × [0, 1000] scale.Categories include wait/observe, positional, gestural, text-input, and system/status actions.
  • Dataset Composition: The dataset comprises 709 multi-step episodes and 2,579 action steps across 29 smartphone applications, with balanced Chinese and English application coverage.Chinese applications contribute 363 episodes and 1,303 steps, while English applications contribute 346 episodes and 1,276 steps.
  • Task Dimensions and Formulation: Five task dimensions cover localization, semantic understanding, cross-modal discrimination, temporal reasoning, and instant response across different cognitive processing requirements.Their episode/step distributions are 20.5%/446, 19.3%/530, 19.9%/514, 22.0%/617, and 18.3%/472, respectively.
  • Annotation Procedure and Quality Assurance: Independent annotation of 100 episodes achieved Cohen’s κ = 0.84, with edge-case disagreements resolved by a third senior annotator through majority vote.Annotators first judged screenshot-only resolvability and then revealed video and audio to assign objective dependency labels.

4 Experiments

Experiments use foundational omni-modal models as proxies to evaluate OmniGUI’s multimodal benchmark mechanisms and establish initial agent baselines. Results show strong static visual performance but substantial difficulties with temporal, auditory, and irrelevant multimodal signals.

  • Experimental Setup: The evaluation uses foundational omni-modal models as direct proxies because dedicated omni-agent frameworks remain nascent.The experiments target benchmark validation and initial performance baselines.
  • Evaluated Models: Eight models are evaluated: four proprietary Gemini models and four open-source models, including Qwen3-Omni, MiniCPM-o 4.5, VITA-1.5, and Baichuan-Omni-1.5.The proprietary models are Gemini 3.0 Pro, Gemini 3.0 Flash, Gemini 2.5 Pro, and Gemini 2.5 Flash.
  • Overall Results: 66.4% EM and 33.1% SR: Gemini 3.0 Pro achieves the highest overall proprietary-model performance, although its success rate remains low.This indicates that multi-step GUI execution with interleaved transient multimodal signals remains difficult for current models.
  • Overall Results: 33.4% EM and 5.2% SR: Qwen3-Omni leads the open-source models, while the remaining open-source models have SR ≤1.1%.The evaluation therefore reveals a substantial capability gap between proprietary and open-source models.
  • Dimension Breakdown: 79.9% Localization EM for Gemini 3.0 Pro exceeds its 59.9% Cross-modal Discrimination and 61.8% Temporal Reasoning EM.Performance is generally higher on static localization than on tasks requiring dynamic temporal and auditory integration.
  • Ablation Studies: 10.5% EM: removing audio and video causes the largest cited drop for Gemini 3 Pro on AV-Critical tasks, while irrelevant full multimodal input lowers Gemini 2.5 Flash’s EM from 49.9% to 40.8%.These ablations support the annotated dependency levels and expose cross-modal interference from task-irrelevant environmental signals.

5 Conclusion and Future Work

OmniGUI introduces a step-level benchmark with continuous interleaved image, video, and audio inputs for evaluating GUI agents in omni-modal smartphone environments. Results show strong performance on static visual tasks but significant degradation with synchronous temporal and auditory signals, while the offline protocol leaves autonomous error recovery unevaluated.

  • Contributions: OmniGUI is the first step-level benchmark for GUI agents in omni-modal smartphone environments, providing interleaved images, video clips, and synchronous audio at every action step.The benchmark contains 709 expert-demonstrated episodes and 2,579 action steps.
  • Findings: Current omni-modal models perform competently on static visual tasks but degrade significantly when action prediction requires synchronous temporal and auditory signals.The benchmark establishes initial baselines using foundational omni-modal models as agent proxies.
  • Findings: Ablations identify cross-modal interference and performance degradation from processing dual-audio streams as operational bottlenecks.Qualitative analysis also shows auditory neglect after a transient acoustic change and spatial grounding failures in coordinate prediction.
  • Limitations and Future Work: The offline step-level protocol isolates per-step perception-to-action capabilities and supports deterministic reproducibility but does not evaluate recovery from compounding errors in autonomous end-to-end rollouts.Future work could extend the evaluation protocol toward autonomous, interactive settings.

A Experimental Details · A.1 Unified Prompt Templates and Input Structure

The experiments use a unified, model-agnostic prompt protocol built from a two-part system/user structure with interleaved historical and current multimodal inputs. Ablations deterministically modify the input payloads and prompt instructions to isolate modality availability and dual-audio processing.

  • A Experimental Details: All evaluated models use an identical prompt structure without model-specific prompt engineering.This standardizes the evaluation protocol across models.
  • A.1 Unified Prompt Templates and Input Structure: The baseline prompt consists of a System Prompt and an interleaved User Message.The User Message combines task context, action history, screenshots, audio, and video.
  • A.1 Unified Prompt Templates and Input Structure: The output protocol requires exactly one valid JSON object, including explicit actions for waiting, completion, and impossible tasks.The rules require agents to act from the latest screenshot and still emit a valid JSON action when uncertain.
  • A.1 Unified Prompt Templates and Input Structure: The task description dynamically uses the English or Chinese instruction corresponding to the target application’s native environment.This language selection is part of the interleaved User Message design.
  • A.1 Unified Prompt Templates and Input Structure: The User Message orders optional historical text and imagery before current-step video, environment audio, the current screenshot, and prompt text.The prompt text includes the task goal, current step index, action history, and an instruction to output the next action as JSON.
  • A.1 Unified Prompt Templates and Input Structure: Modality ablations physically omit audio, video, or both from the User Message and minimally adjust the System Prompt to describe the remaining modalities.These modifications are applied deterministically at the input level.
  • A.1 Unified Prompt Templates and Input Structure: Dual-audio experiments insert a spoken-instruction .wav file immediately before the text prompt while replacing the textual task description with a static placeholder.The System Prompt adds a directive to listen to the voice instruction audio.

A.2 Model Configurations and Hyperparameters

The evaluations used greedy decoding to produce deterministic, reproducible outputs and assess models’ most confident decision boundaries. Hyperparameter settings varied by model type, with proprietary API parameters set to zero where supported and some open-source models evaluated using official defaults.

  • Greedy decoding was enforced across all evaluations for deterministic, reproducible outputs and assessment of the models’ most confident decision boundaries.
  • Proprietary API models used parameters explicitly set to zero where supported.
  • Certain open-source models, including VITA and Baichuan-Omni-1.5, were evaluated under their officially recommended default server interfaces.

B Dataset Construction Details

This section details OmniGUI’s dataset statistics and application-level composition. Table 7 distributes episodes and action steps across five task dimensions and three multimodal dependency levels for 29 smartphone applications.

  • Dataset Statistics: Table 7 provides a comprehensive application-level breakdown of OmniGUI’s dataset statistics.The table details the exact distribution of episodes and action steps.
  • Application Coverage: The breakdown covers all 29 evaluated smartphone applications.
  • Dataset Dimensions: Episodes and action steps are distributed across five task dimensions and three multimodal dependency levels.

B.1 Data Format Example

OmniGUI organizes each application into episode metadata, step-level multimodal assets, and chronological execution traces. The format records task dimensions, multimodal dependency levels, synchronized media, and precise actions for reproducible evaluation.

  • Dataset Breakdown: The dataset table reports episode and step volumes with distributions across five task dimensions and three multimodal dependency levels.The dependency levels are AV-Critical, AV-Supportive, and AV-Present.
  • Directory Structure: Each application directory separates five task-dimension JSONL files, a global step index, and media folders containing episode-level assets.The five dimensions are localization, semantic understanding, cross-modal discrimination, temporal reasoning, and instant response.
  • Step-Level Execution Trace: Each episode media folder stores interleaved video clips, audio tracks, screenshots, and a JSON action trace for every step.The trace records chronological waiting and execution states alongside target bounding boxes.
  • Episode-Level Metadata: Episode metadata files store bilingual task descriptions that map directly to the unified evaluation prompt’s task_description placeholder.The TED temporal-reasoning example includes instructions requiring actions at precise visual or auditory moments.
  • Step-Level Execution Trace: Execution records encode synchronized media paths, image dimensions, action types, and touch coordinates, including NONE waits and TAP actions.An example uses 720×1600 screenshots and a TAP bounding box represented by two coordinate pairs.

C Additional Experimental Results · C.1 Model Capability Fingerprints

The supplementary results provide visual analyses of baseline evaluations, profiling model capabilities across cognitive dimensions, multimodal dependencies, and application environments. Figure 4 presents Exact Match capability fingerprints across five task dimensions and three multimodal dependency levels.

  • C Additional Experimental Results: The supplementary section provides visual analyses that offer a granular view of baseline model capabilities.The analyses span cognitive dimensions, multimodal dependencies, and application environments.
  • C Additional Experimental Results: The analyses characterize model performance across multiple cognitive dimensions.These dimensions are part of the supplementary capability profiling framework.
  • C Additional Experimental Results: The analyses examine model behavior under different multimodal dependencies.This provides a multimodal perspective on the baseline evaluations.
  • C Additional Experimental Results: The supplementary visualizations compare capabilities across application environments.Application environments are included alongside cognitive dimensions and multimodal dependencies.
  • C.1 Model Capability Fingerprints: Figure 4 presents multidimensional capability fingerprints for the evaluated models using Exact Match (EM).The figure is designed to visualize specific performance profiles.
  • C.1 Model Capability Fingerprints: The capability-fingerprint analysis reports performance across five predefined task dimensions.This is one of the two perspectives used in Figure 4.
  • C.1 Model Capability Fingerprints: The capability-fingerprint analysis also reports performance across three multimodal dependency levels.This is the second perspective used in Figure 4.

C.2 Performance Breakdown by Application

Gemini 3.0 Pro’s performance varies across the 29 evaluated smartphone applications. Figure 5 reports application-level Exact Match, Goal Progress, and Success Rate against overall benchmark averages.

  • Application-level performance: Figure 5 disaggregates Gemini 3.0 Pro’s Exact Match, Goal Progress, and Success Rate across 29 evaluated applications.The applications are sorted by Exact Match score, with sample volume shown on the right-hand axis.
  • Application-level performance: 63.6% Exact Match and 43.6% Goal Progress are the overall benchmark averages used as application-level reference lines.Vertical dotted lines identify application environments with above-average or below-average difficulty.
  • Capability fingerprints: Models perform better on static Localization than on Temporal Reasoning and Cross-modal Discrimination across the five operational dimensions.This capability pattern is shown in the evaluated models’ radar-chart fingerprints.
  • Capability fingerprints: Performance declines monotonically as tasks transition from AV-Present to AV-Critical multimodal dependency levels.The radar charts visually validate the necessity of multimodal perception.
Loading 2605.18758v1…