PaperlayerView exact revision on arXiv

Source-linked AI summary

Focused Omni-Modal GUI Agents

Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang, Wei Yang Bryan Lim

arXiv:2610.26203v1cs.CV

TL;DR

In streaming omni-modal GUI interaction, relevance is sparse within steps and shifts across them, while action feedback can compound misreads. FocusGUI adds a step-specific focus signal to a frozen backbone without changing its input. Across four backbones, it improves Type Match and Exact Match over frozen inference and Type Match over the strongest enhancement method.

  • Problem

    In multi-step omni-modal GUI interaction, relevance is transient across steps and sparse within each observation, while decoder misreads can compound as actions feed back into later observations.

    Source: §1 · p. 2

  • Method

    FocusGUI scores perceptual channels against a step-specific reference, compresses prioritized evidence into a compact latent, and injects it as a residual into the frozen decoder while training on action supervision alone.

    Source: §1 · p. 2, §2 · pp. 3–4

  • Results

    +33.2 Type Match points over direct inference on average across four backbones; FocusGUI also adds 30.1 Type Match points over the strongest baseline at each backbone.

    Source: §3.2 · p. 6

  • Takeaways & Limitations

    The current formulation computes channel priorities from each step’s available observation; extending it to anticipate evidence needed in subsequent steps is identified as future work.

    Source: B Discussion and Limitations · p. 15

    The experiments span four omni-modal backbones and five interaction categories, with broader computer-use settings left for future work.

    Source: B Discussion and Limitations · p. 15

Abstract

from arXiv · show

Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.

1 Introduction

In multi-step omni-modal GUI interaction, decision-relevant evidence is sparse within each step and can shift between steps, so decoder misreads can compound as actions feed back into later observations. FocusGUI addresses this by adding a step-specific focus signal without altering the observation or frozen backbone, and improves action matching across backbones.

1 Introduction

  • Relevance is transient across steps and sparse within them, and executed actions feed back into observations so misreads can compound into trajectory drift.The paper calls this compounding failure Sequential Distraction.Source: §1 · p. 2
  • Spatial cropping cannot select audio, video, or device-state signals and may remove screen context needed to ground coordinates and text arguments.The proposed alternative is to guide how the decoder reads the observation rather than change what it receives.Source: §1 · p. 2
  • FocusGUI scores perceptual channels against a step-specific decision reference, compresses prioritized evidence into a focus latent, and adds it as a zero-initialized residual at the action boundary.The frozen backbone pathway remains intact, and training uses next-action supervision without modality-relevance labels.Source: §1 · p. 2, Figure 2 · p. 3
  • Across four omni-modal backbones on average, FocusGUI raises Type Match from 31.0% to 64.2% and Exact Match from 11.2% to 33.5% over the frozen backbone.It also reaches 1.9× the Type Match of the strongest enhancement method.Source: §1 · p. 2
  • The paper distinguishes the evidence an agent receives from how its decoder reads that evidence, motivating a solution that guides attention while preserving the input.Its contribution summary says spatial refinement alone cannot satisfy both action selection and argument grounding.Source: §1 · pp. 2–3

2 Method

FocusGUI preserves the full omni-modal observation while learning a compact, step-specific signal that guides the frozen decoder’s read at the action boundary.

Problem Formulation and Context-Preserving Focus

  • FocusGUI preserves the complete observation and adds a compact focus residual to existing decoder embeddings at the action boundary.The original pathway retains multimodal context; the auxiliary pathway guides how it is emphasized for the next action.Source: §2 · pp. 3–4

Step-Adaptive Evidence Organization

  • Each step, FocusGUI evaluates separable perceptual channels against a joint semantic context and recomputes their relative priorities.This lets channel emphasis change as the task evolves, without carrying priority state across steps.Source: §2 · p. 4

Compact Focus and Residual Alignment

  • The prioritized representation is projected into a lower-dimensional focus latent and aligned to the decoder’s native embedding space.It conditions emphasis rather than reconstructing the observation or independently encoding execution details.Source: §2 · p. 4
  • Zero-initialized residuals preserve the original embeddings at training start, and FocusGUI neither appends textual hints nor removes multimodal inputs.The focus signal is added to existing prompt positions immediately before action generation.Source: §2 · p. 4

Learning Focus from Action Supervision

  • FocusGUI learns its channel prioritization, compression, and alignment from executable-action supervision without modality-relevance labels.The action-token loss updates the adapter while the backbone and decoder remain frozen.Source: §2 · pp. 4–5

3 Experiments

Experiments evaluate FocusGUI against frozen omni-modal backbones and enhancement baselines on OmniGUI, then examine component contributions and performance across interaction depth. FocusGUI improves overall match metrics over direct inference and strong baselines, with gains that widen as trajectories lengthen.

3.1 Experimental Setup

  • The evaluation uses 708 OmniGUI episodes across 29 applications, with 2,572 action steps pairing screenshots with synchronous audio, video, and action history.Four frozen backbones and four enhancement baselines are compared under the same observations and decoding procedure.Source: §3.1 · pp. 5–6

3.2 Main Results

  • +33.2 TM, +22.3 EM, and +6.6 GP points over direct inference; FocusGUI also leads the strongest baseline by 30.1, 23.6, and 8.0 points, respectively, averaged across four backbones.It records the best TM and EM in all 24 backbone-category cells.Source: §3.2 · p. 6
  • Screen-refinement baselines average +0.9 TM points but reduce EM by 28% and GP by 35% in relative terms, in 12 of 12 cases for each metric.Narrowing the screenshot can highlight the action primitive while discarding context needed for coordinates and text, and it cannot access audio, video, or device state.Source: §3.2 · p. 6
  • Per-category TM gains range from 37.3 points on Temporal tasks to 27.8 on Instant tasks, while overall TM spread across backbones narrows from 35.6 to 13.0 points.EM variation instead reflects backbone-specific argument-generation capacity.Source: §3.2 · p. 6

3.3 Ablation Study

  • Removing action-sufficient projection, residual alignment, step-adaptive prioritization, and channel decomposition lowers TM by 0.51, 1.20, 4.27, and 5.39 points, respectively, and GP by 1.38, 5.38, 5.93, and 5.29 points.Each component has a proportionally larger effect on episode progress than on single-step action type.Source: §3.3 · p. 7, Table 2 · p. 7

3.4 Performance Across Interaction Depth

  • Qwen2.5-Omni TM falls from 64.79 at the first step to 32.79 beyond the third, while FocusGUI retains 86.8% and its EM rises from 38.73 to 46.72.EM gains widen from +22.54 to +35.25, with all intervals excluding zero; Figure 3 bins steps by episode index and shows 90% episode-clustered bootstrap intervals.Source: §3.4 · p. 7, Figure 3 · p. 7
  • For Instant-response tasks, FocusGUI is statistically indistinguishable from the backbone at the first two steps, then gains +28.57 and +46.67 TM points at depths 2 and 3+.The widening margin persists when restricting analysis to episodes with at least four steps.Source: §3.4 · p. 8
  • Interaction-state priority falls from 30.9% at opening to 7.7% at closing, while audio rises from 13.4% to 18.8% and video from 11.3% to 18.3%.The screen remains below uniform allocation at every phase; the learned ordering uses action annotations without modality-relevance supervision.Source: §3.4 · p. 8, Figure 4 · p. 8

4 Related Work

GUI agents have moved toward long-horizon interaction over evolving omni-modal observations, but how to organize that evidence for the next action remains open. FocusGUI complements GUI grounding by extending selective perception beyond screen regions to step-dependent evidence across the complete observation.

4.1 GUI Agents in Omni-Modal Environments

  • OmniGUI places actions within interleaved screen, video, audio, and interaction-history observations, extending GUI interaction toward an evolving omni-modal state.Earlier work commonly organized observations around the current screenshot, with prior actions or screens as auxiliary context.Source: §4.1 · p. 9
  • Omni-modal models support unified or streaming perception, but how to organize their evidence for the next GUI action remains open.FocusGUI addresses this by learning which evidence should shape the current action and how to deliver it to a frozen decoder.Source: §4.1 · p. 9

4.2 Perception and Grounding for GUI Agents

  • GUI representation learning, grounding data, screen parsing, and visual-agent methods improve interface understanding and action localization.These approaches include GUI-specific representations, explicit interface structure, and reduced screen tokens jointly trained for grounding and action generation.Source: §4.2 · pp. 9–10
  • Selective visual search and active perception suppress irrelevant screen content, but primarily control which region of the current screen receives attention.FocusGUI extends focused perception to step-dependent representation construction across the complete observation, including multiple information sources.Source: §4.2 · p. 10

5 Conclusion

The conclusion frames transient, sparse relevance as a challenge in multi-step omni-modal GUI interaction and presents FocusGUI as a lightweight, action-supervised adapter for frozen backbones. It reports improvements across four backbones and identifies anticipating future channel relevance as a possible extension.

5 Conclusion

  • FocusGUI addresses compounding errors from transient, sparse relevance by prioritizing perceptual channels into an action-sufficient latent for a frozen backbone.It uses action annotations alone and delivers the latent through a zero-initialized residual at the action boundary.Source: §5 · p. 10
  • Evaluations across four backbones show improved Type Match over existing enhancement methods and maintained accuracy at deeper interaction steps.The conclusion states that historical context typically degrades at those deeper steps.Source: §5 · p. 10
  • Future work could extend current-observation channel prioritization with anticipatory signals about evidence that may matter in subsequent steps.The paper suggests this may help mitigate trajectory drift earlier in an interaction sequence.Source: §5 · p. 10

A Notation

B Discussion and Limitations

FocusGUI computes channel priorities from the observation at each step, with broader computer-use settings identified as a next step. The capacity analysis reports a flat parameter grid around the default, apart from a configuration where compression removes needed information.

B Discussion and Limitations

  • Current channel priorities use only the observation available at each step; broader computer-use settings remain a next step for evaluation.The formulation is described as simple and model-agnostic, and the experiments span four backbones and five interaction categories.Source: B Discussion and Limitations · p. 15
  • Anticipating which evidence may matter in subsequent steps is identified as a natural extension of the current formulation.Source: B Discussion and Limitations · p. 15

C.1 Capacity surface.

  • The capacity surface is flat across most measured settings, while dz = 32 is the configuration where compression removes information needed for the action.The default lies in the flat region rather than at a parameter-dependent peak.Source: §C.1 · p. 15

C.2 Interaction depth across backbones.

Across evaluated backbone families, FocusGUI's accuracy holds as frozen-backbone accuracy declines with interaction depth, widening the margin. A parameter-matched control also favors dynamic prioritization over fixed concatenation, especially on Exact Match and Goal Progress.

C.2 Interaction depth across backbones.

  • Figure 6's grid is flat around the default K = 4, dz = 64; circles mark measured settings, while interpolated regions are unmeasured.Source: Figure 6 · p. 16
  • Across backbones spanning 13.98 to 36.02 frozen Overall TM, FocusGUI holds accuracy as frozen models decline with depth, widening the margin in every case.On Exact Match, the FocusGUI curve is flat or rising where the frozen curve falls; the effect transfers across backbone families without architecture or training-recipe changes.Source: §C.2 · pp. 15–16, Figure 7 · p. 16, Figure 8 · p. 17

C.3 Prioritization, not capacity.

  • +3.54 Exact Match and +3.44 Goal Progress, with +0.98 Type Match, under dynamic prioritization versus a parameter-matched fixed-concatenation baseline.Unlike uniform pooling, fixed concatenation preserves each channel in its own slot and lets the downstream projection learn step-independent weights; the control isolates per-step scoring.Source: §C.3 · p. 16

C.4 Paired Error Correction

FocusGUI improves paired action correctness across all five task categories, with the largest net improvement in temporal reasoning.

C.4 Paired Error Correction

  • 153 predictions are corrected versus 13 regressed overall, and corrections outnumber regressions in all five task categories.The largest net improvement is in temporal reasoning.Source: §C.4 · p. 17
  • Temporal reasoning has the largest net improvement, indicating particular effectiveness on decisions over evolving observations.Source: §C.4 · p. 17
  • Corrections consistently outnumber regressions across all five categories.Source: §C.4 · p. 17

D More Implementation Details

The implementation keeps the task prompt and action vocabulary fixed while emitting structured actions through the frozen language-model head. Type Match evaluates the action primitive, while Exact Match evaluates its arguments.

D. More Implementation Details

  • The 13-primitive action space includes pointer, swipe, text-entry, navigation, termination, and observation-or-wait actions.Coordinates are normalized to [0, 1000], and the frozen language-model head emits actions as text rather than classifying screen regions on a grid.Source: §D.1 · p. 17
  • Type Match reads the action primitive, while Exact Match reads its arguments from the generated JSON object.Source: §D.1 · p. 17
  • The system prompt fixes the agent role and requires one valid JSON object per step, without explanations or extra text.Source: §D.1 · p. 18
  • The action instructions specify coordinate order and normalization, and reserve NONE for observation or waiting and TASK_COMPLETE for an achieved goal.Source: §D.1 · p. 19

D.2 Training target and inference.

Training supervises serialized actions while masking prompt tokens from the loss; inference generates actions with the original frozen language-model head. The step template preserves multimodal inputs and action history, ending at the action boundary where FocusGUI applies its residual.

D.2 Training target and inference.

  • Training loss is computed only over assistant action tokens, with system and user tokens masked; inference uses greedy decoding with beam size one.The target action is serialized as a compact JSON string during training and absent at inference.Source: §D.2 · p. 19
  • FocusGUI leaves the semantic prompt and observation inputs unchanged, injecting selected evidence as continuous latent residuals at the assistant action boundary.Source: §D.2 · p. 19, §D.3 · p. 19

D.3 Step template.

  • The step template orders history screenshots, video, audio, and the execution screen before the goal, step index, and text-rendered action history.The execution screen is the most recent visual input, and the prompt ending fixes the action boundary for residual application.Source: §D.3 · p. 19

D.4 Experiment Details

The experiments use episode-disjoint splits and small backbone-specific adapters, with identical inference inputs and decoding for raw and FocusGUI models. The algorithm adds a step-specific residual without updating the frozen backbone, and case studies cover auditory-temporal and spatial-semantic GUI tasks.

D.4 Experiment Details

  • FocusGUI adapters contain 1.15M–1.45M parameters, with 1.445M trainable parameters against 8.932B frozen Thinker parameters on Qwen2.5-Omni.The Qwen2.5-Omni adapter is 0.0162% trainable parameters; adapters are separate for each backbone and experiments use two NVIDIA L20 GPUs.Source: §D.4 · pp. 19–20
  • The episode-disjoint train, validation, and test splits contain 1,807, 257, and 508 steps from 495, 71, and 142 trajectories, respectively.Normalization statistics are fit on training data only; validation selects checkpoints, and the test split is evaluated once after selection.Source: §D.4 · p. 19

D.5 Controlled inputs.

  • Raw backbones and FocusGUI receive identical prompts, observations, action histories, schemas, and decoding; neither receives demonstrations or modality hints.The parameter-matched control removes per-step scoring, isolating the content of the residual from prompt length, in-context examples, and decoding changes.Source: §D.5 · p. 20

E Algorithm

  • At each step, the forward pass scores perceptual channels against a decision reference, compresses selected evidence, and adds a zero-initialized residual before the action boundary.The prompt is not modified, backbone parameters remain frozen, and gradients flow through the decoder to FocusGUI parameters.Source: E Algorithm · p. 21, §D.5 · p. 21

F Case Study

  • The case studies span auditory-temporal monitoring and multi-step spatial-semantic navigation through a dynamic interface.The examples describe shifting attention to audio instead of static visual saliency and sequentially selecting nested filter options; the paper presents them as evidence of generalization across contrasting GUI challenges.Source: F Case Study · p. 21, Figure 11 · p. 22, Figure 12 · p. 22

E Algorithm

  • Training uses action supervision with the encoder, decoder, and language-model head frozen; inference executes each generated action, which re-enters the next observation.The algorithm uses the original language-model head with greedy decoding and beam size one.Source: F Case Study · p. 23

G Theoretical Analysis

The analysis traces how FocusGUI’s latent affects action logits through two pathways with distinct roles: attention re-routing and content changes. It relates those pathways to action primitives and arguments and bounds the adapter’s control to a low-dimensional family.

G Theoretical Analysis

  • The latent reaches action logits through two pathways: one can select the action primitive, while the other is the only route for correcting arguments.The analysis identifies these as distinct ways of changing what the decoder reads and where it reads.Source: G Theoretical Analysis · p. 23
  • The adapter steers the frozen computation through a family of dimension at most dz, while the network’s response remains fixed by Θ0.This bounds the control signal’s dimension, not the information available through the backbone.Source: G Theoretical Analysis · p. 23
  • From layer 2 onward, query steering redistributes attention across the observation, although the residual is confined to K positions.At layer 1, only key displacement is active and reweights attention within Jt.Source: G Theoretical Analysis · p. 24
  • At every layer, evidence-position content is independent of γ, so the adapter guides attention without changing that content.The analysis states this for positions i < bt − K.Source: G Theoretical Analysis · p. 24
  • Argument corrections arise through pathway (A), which re-routes attention onto correct evidence; pathway (V) supplies content for primitive changes.Replacing the additive injection with replacement would overwrite recent context positions used as evidence.Source: G Theoretical Analysis · p. 24

Related papers

Loading 2610.26203v1…