PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Tool-Mediated Interaction Failures in Agentic GUI Software: An Empirical Study

Yuchen Shao, Wenjing Liu, Yuheng Huang, Mengqian Xu, Ziqun Bao, Long Yang, Lei Ma, Ting Su, Chengcheng Wan

arXiv:2610.31698v1cs.SE

TL;DR

Tools that observe and operate live GUIs can leave an LLM agent’s view misaligned with the interface, while partial observations are one stated source of incomplete task views. The paper analyzes confirmed failures and introduces a defect library and static checker, reporting the checker’s precision and recall on an additional corpus.

  • Problem

    Tools connecting LLM agents to live GUIs can leave the agent’s view of the task silently divergent from the interface, while partial observations may omit relevant GUI elements.

    Source: A. Motivation · p. 1

  • Method

    The paper constructs MALVA from confirmed tool-mediated failures and operationalizes the defect taxonomy with MALVAGUARD, a pre-deployment static contract checker.

    Source: §VIII · p. 10

  • Results

    MALVAGUARD reaches 57.3% precision and 76.8% recall on an additional 20-application corpus.

    Source: §VIII · p. 10

  • Takeaways & Limitations

    The authors present MALVA and MALVAGUARD as a starting point for improving the reliability of agentic GUI software.

    Source: §VIII · p. 10

    The findings may not generalize beyond the open-source corpus, studied platforms, and common-use scenarios, which may omit specialized workflows or closed-source constraints.

    Source: §VI · p. 10

Abstract

from arXiv · show

Large language model (LLM) agents increasingly complete tasks through live graphical user interfaces (GUIs). Because the model cannot directly operate the GUI, these systems often rely on tools to capture screenshots or UI trees, translate model outputs into GUI actions, and report whether the interface changed. We study failures where this tool-mediated interaction makes the agent's view of the task diverge from the live GUI. We construct Malva, a defect library with 335 confirmed real-world failures from 40 open-source applications. These failures fall into 11 defect patterns rooted in state representation, state transition, and synchronization problems. They are widespread: 90% of the studied applications contain more than one defect type. They are also often silent: incorrectness appears in 86.6% of failures, including 119 cases that produce wrong behavior without a fail-stop signal. We operationalize this taxonomy with MalvaGuard, a pre-deployment static contract checker that reaches 57.3% precision and 76.8% recall on an additional 20-application corpus.

I. INTRODUCTION

Agentic GUI software uses tools to observe interfaces, execute actions, and report feedback, but this mediation can silently separate the agent’s view from the live GUI. The introduction frames partial observations, weak action boundaries, delayed feedback, and neglected tool-layer failures as central reliability challenges.

I. INTRODUCTION

  • The UI-TARS-desktop example shows a send click reaching the agent’s overlay instead of the button because the tool exposes coordinates without checking which window has focus.The message is never sent, despite the preceding typing action succeeding.Source: A. Motivation · p. 1
  • Partial GUI observations can omit obscured, off-screen, absent-from-tree, or non-actionable elements, leaving the agent to plan from an incomplete view.Source: A. Motivation · p. 1
  • Weak action boundaries leave tools’ accepted parameters, GUI preconditions, return formats, and stopping or retry rules unspecified, allowing malformed calls and non-progressing retries.The same gaps can also lead to context overflow or actions targeting the wrong element.Source: A. Motivation · pp. 1–2
  • Delayed feedback can make reported tool status diverge from the live interface when GUI changes, backend timeouts, or expired resource handles intervene.Source: A. Motivation · p. 2
  • Prior work often treats the tool layer as invisible infrastructure or studies model and agent behavior, leaving tool-mediated failures underexamined.The paper addresses this gap by systematically studying failures introduced by tools connecting agents to GUI environments.Source: A. Motivation · p. 2

B. Contribution

The paper studies tool-mediated interaction failures across agentic GUI software, where observations, actions, and feedback must remain aligned with the live environment. It reports widespread defects and presents MALVA and MALVAGUARD as a defect library and pre-deployment checker.

B. Contribution

  • The study confirms 335 failures from around 2,000 issue reports across 40 GitHub projects and releases MALVA with their root causes and impacts.Source: B. Contribution · p. 2
  • 90% of studied applications contain multiple defect types, and the study identifies 11 patterns rooted in state representation, transition, and synchronization problems.MALVAGUARD reaches 57.3% precision and 76.8% recall on an additional 20-application corpus.Source: B. Contribution · p. 2

A. Agentic GUI Software

  • Agentic GUI software bridges LLM agents and live web, desktop, or mobile environments through observation, action, and feedback tools.Observation tools provide GUI evidence; action tools translate selected actions into backend effects; feedback tools report status and visible effects.Source: A. Agentic GUI Software · p. 2

B. Tool-Mediated Interaction Loop

  • A tool-mediated cycle transforms both environment and agent state, while tools may also update local caches, parsed snapshots, and backend handles.The four phases observe and ground GUI state, select an action using agent context, execute the operation, and report effects back.Source: B. Tool-Mediated Interaction Loop · pp. 2–3

C. Interaction Failures

The paper defines interaction failures as misalignment between agent and environment state, including both hard failures and soft failures. It builds MALVA from a selected corpus of open-source agentic GUI applications.

C. Interaction Failures

  • An interaction failure occurs when tools fail to keep agent state aligned with environment state, including crashes, exceptions, stalled loops, unproductive repetition, or wrong-task completion.A defect is an invalid state accepted or executed without the necessary validation or effect check.Source: C. Interaction Failures · p. 3

A. Application Selection

  • The study selects 40 GitHub applications whose repositories implement executable real-world agent loops, concrete GUI observation and action tools, and sufficient implementation and issue evidence.The researchers reviewed about 200 additional repositories and stopped collection after finding no major differences from the corpus.Source: A. Application Selection · p. 3
  • The corpus spans web, desktop, and mobile agents, with varied languages, reasoning models, GUI observation methods, and grounding mechanisms.The applications are 40% web, 43% desktop, and 18% mobile; most use Python or TypeScript, and GUI observation ranges from screenshots to structural representations or both.Source: A. Application Selection · p. 3
  • The applications average 71,687 lines of code, while 63% are less than one year old.The reported line-count median is 24,978, with an interquartile range of 2,910 to 57,146.Source: A. Application Selection · p. 3

B. Issue Collection

The study screens issue reports from 40 applications and combines issue-history review with task reproduction to confirm failures. For reproducible open issues, it defines executable tasks and runs them repeatedly with automated evidence logging.

B. Issue Collection

  • About 2,000 issue reports are screened with keyword filters and an LLM-assisted classifier, followed by manual review that retains 208 candidates.The reports come from the 40 applications; dependency, infrastructure, and ambiguous-prompt reports are excluded.Source: B. Issue Collection · p. 3

C. Task Execution

  • The 208 candidates split evenly between closed and open issues; closed cases are assessed through discussions, linked pull requests, and commits.Open cases are reproduced when the needed environment and credentials are available.Source: C. Task Execution · p. 4
  • Open issue scenarios yield 62 executable tasks, each specifying an initial state, agent instruction, and success oracle under 14 environment-specific scenarios.The tasks cover web, desktop, and mobile agents, with 35, 14, and 13 tasks respectively.Source: C. Task Execution · p. 4
  • The 62 tasks are each run 20 times with a three-minute timeout, producing 1,240 runs and about 3,000 logged agent interactions.Logs capture screen recordings and DOM snapshots, agent prompts and actions, tool calls and returns, and environment feedback.Source: C. Task Execution · p. 4

D. Defect Identification

The study validates 335 interaction failures across 40 applications and classifies them into 11 recurring tool-layer patterns. These failures vary by phase and setting, and incorrect behavior is often silent.

D. Defect Identification

  • Manual confirmation required converging evidence, excluded failures limited to unstable environments or unsupported setups, and iterative labeling reached 95% initial agreement (Fleiss’ κ = 0.95).Four authors refined the patterns; three independently categorized failures, with a fourth resolving remaining disagreements, over about seven person-months.Source: D. Defect Identification · p. 4

A. Overview

  • The study classifies 335 defects across 40 applications into 11 patterns, organized by workflow phase and the violated contract among agent state, tool behavior, and live GUI state.The 99 initial confirmed tool-layer mistakes were expanded by inspecting the other 39 applications and reproducing matching failures.Source: A. Overview · pp. 4–5, D. Defect Identification · p. 4
  • State Representation Failure accounts for 26.3% of defects, State Transition Failure for 55.5%, and Synchronization Failure for 18.2%.These categories capture mismatches in GUI evidence, invalid or unguarded interaction advances, and feedback or lifecycle status that diverges from the environment.Source: A. Overview · p. 5
  • Progress Check Failure, Tool Schema Mismatch, and Lifecycle Propagation Failure occur in 65%, 63%, and 58% of studied applications, respectively; together with Context Budget Overflow, they account for 63.0% of failures.P2 alone contributes 31.0% of failures, where loops proceed before checking progress, budget, or escalation conditions.Source: A. Overview · p. 5
  • Across settings, model-visible state may not match the live GUI; incorrectness appears in 86.6% of confirmed failures, including 119 cases of wrong behavior without a fail-stop signal.Desktop agents account for 56.1% of confirmed failures; screenshot-based agents fail more in observation and execution, while structure-only agents fail more in P2.Source: A. Overview · p. 5

B. Phase 1: Observation Defects

Observation and grounding failures arise when tools provide absent, misbound, or incomplete GUI evidence. The resulting agent state can diverge from the live interface before an action is taken.

B. Phase 1: Observation Defects

  • 88 of 335 failures arise when GUI evidence used to build Stateagent does not reflect the live GUI, and tools fail to verify it against the target the executor will act on.The defects include missing, wrong, stale, or incomplete evidence from screenshots, structural snapshots, detected elements, and grounding results.Source: B. Phase 1: Observation Defects · p. 5
  • Observation Hallucination affects 13 applications (33%), averaging 2.69 cases each, and includes fake completion claims and actions against nonexistent targets.AppAgent issues tap(70) when only 66 detected elements exist, causing an IndexError.Source: B. Phase 1: Observation Defects · p. 5, Figure 5 · p. 5
  • Target Grounding Error affects 38% of applications, as rigid geometry checks, coordinate drift, or semantic confusion can turn a valid target into an invalid or wrong locator.LaVague rejects a widget whose center is covered even though other clickable regions remain exposed; UI-TARS-desktop illustrates coordinate-rounding drift, and computer use ootb confuses taskbar icons with browser launchers.Source: B. Phase 1: Observation Defects · pp. 5–6, Figure 6 · p. 6
  • State Exposure Gap affects 18% of applications, with a median of two cases, when tools omit GUI state relevant to the next action.Examples include hidden click ownership, stale screen dimensions, horizontally scrollable content, and dialogs or loading states that conceal actionable UI.Source: B. Phase 1: Observation Defects · p. 6

C. Phase 2: Reasoning & Planning Defects

P2 failures occur when the tool loop advances without checking whether the next step is valid, progressing, within budget, or suitable for escalation. This phase contributes 104 of 335 failures.

C. Phase 2: Reasoning & Planning Defects

  • Progress Check Failure affects 26 applications (65%), averaging 3.08 cases each, when repeated unchanged outcomes do not trigger stopping, recovery, or a strategy change.In computer use ootb, screenshotting collapses a dropdown before selection, creating an indefinite open-screenshot-close cycle without a dropdown-open check or repeated-action detector.Source: C. Phase 2: Reasoning & Planning Defects · p. 6, Figure 7 · p. 6
  • Escalation Boundary Failure affects 5% of applications, with a median of 1.5 cases, when security checkpoints are returned as ordinary page states rather than stop signals.WebVoyager continues redundant actions after triggering CAPTCHA verification until its step budget is exhausted.Source: C. Phase 2: Reasoning & Planning Defects · p. 6, Figure 8 · p. 6
  • Context Budget Overflow affects 48% of applications, with 21 cases, when unpruned observation, extraction, or history payloads exceed model-interface limits.browser-use-typescript can pass over 530,000 tokens against a 128,000-token limit; the tool output is accepted without a size check before model invocation begins.Source: C. Phase 2: Reasoning & Planning Defects · p. 7
  • P2 contributes 104 of 335 failures, or 31.0%, with missing guards for payload size, non-progressing retries, and escalation boundaries.The study recommends bounding output before model invocation, detecting non-progressing states before retrying, and surfacing security barriers as stop signals.Source: C. Phase 2: Reasoning & Planning Defects · pp. 6–7

D. Phase 3: Execution Defects

P3 execution failures account for 82 of 335 cases, arising when tools accept or execute actions without validating schemas, parameter meaning, or readiness. These failures can occur even when a call is parseable.

D. Phase 3: Execution Defects

  • P3 execution tools account for 82 of 335 failures when actions are accepted, translated, or executed without checks for validity, parameter correctness, or readiness.The paper recommends validating schemas, rejecting unresolved placeholders, and checking GUI and backend readiness before execution.Source: D. Phase 3: Execution Defects · pp. 7–8
  • Tool Schema Mismatch affects 63% of applications, with a median of two cases, when parsers or registries cannot normalize calls into registered actions with expected fields and types.Examples include Markdown-fenced JSON rejected by screenAgent, unregistered action names from Windows-Use, and pywinassistant crashing on coordinates without commas.Source: D. Phase 3: Execution Defects · p. 7
  • Parameter Semantics Error affects 25% of applications, averaging 1.70 cases per affected application, when adapters change a parseable call’s meaning before execution.Examples include autoMate replacing a requested 200-pixel scroll with 500 pixels and actor-web-automation-agent forwarding an unresolved CAPTCHA placeholder.Source: D. Phase 3: Execution Defects · p. 7
  • Missing Execution Preconditions affects 33% of applications, with a median of two cases, when valid calls reach missing resources or GUI targets that are not ready.Examples include cua accessing a VM before provisioning finishes and Surf issuing terminal commands before launching or focusing the terminal.Source: D. Phase 3: Execution Defects · pp. 7–8, Figure 10 · p. 7

E. Phase 4: Environment Feedback Defects

Phase 4 failures arise when feedback reports success or propagates lifecycle state before the GUI or its resources are stable and usable. They span premature observations and stale or misrouted lifecycle signals.

E. Phase 4: Environment Feedback Defects

  • 61 of 335 failures arise across feedback patterns that report success or propagate state before the GUI or resource is stable and usable.These patterns concern feedback tools closing the loop between tool actions and the next agent step.Source: E. Phase 4: Environment Feedback Defects · p. 8
  • Temporal Feedback Mismatch appears in 8 applications (20%), averaging 1.38 cases each, when feedback precedes a stable, observable GUI state.An early acknowledgment can leave the next observation showing the pre-action state.Source: E. Phase 4: Environment Feedback Defects · p. 8
  • Lifecycle Propagation Failure affects 23 applications (58%), averaging 2.17 cases each, when crashes or disappearing resources are not propagated as structured errors.Agents may continue using stale handles, as when a destroyed sandbox cannot be extended and later calls cannot recover the task.Source: E. Phase 4: Environment Feedback Defects · p. 8
  • Lifecycle state can also be misrouted: a listener treated an agent-generated Escape keypress as a user-stop signal and terminated the run.The action tool could not mark its key event as internal, so the listener propagated a stop without provenance.Source: E. Phase 4: Environment Feedback Defects · p. 8, Figure 12 · p. 8
  • Feedback tools should verify GUI stability before reporting success and propagate resource lifecycle events as structured errors rather than ordinary continuation signals.These failures are the smallest phase by share, at 18.2%, but can be easy to overlook.Source: E. Phase 4: Environment Feedback Defects · p. 8

V. SOLUTION: MALVAGUARD

MALVAGUARD encodes the defect taxonomy as pattern-based static-analysis rules. It locates tool-environment interaction regions, checks for required guards, groups related warnings, and reports defects with repair guidance.

V. SOLUTION: MALVAGUARD

  • MALVAGUARD encodes the taxonomy as pattern-based static-analysis rules to detect code-level anti-patterns before execution.The checker targets missing guards that fail to align agent state with the live environment.Source: §V · p. 8, Figure 13 · p. 9

A. Stage 1: Defect Detection

  • Stage 1 identifies each interaction region and checks whether a required guard is present for the relevant tool-environment interaction.An interaction region is an enclosing function or method, or a matched prompt/file line with nearby lines; checks look for guards within that region.Source: A. Stage 1: Defect Detection · p. 9
  • Stage 1 groups co-located warnings into one defect only when they share a checker rule and missing-contract type.Warnings from different rules remain separate defects, even within the same component.Source: A. Stage 1: Defect Detection · p. 9

B. Stage 2: Defect Attributing

  • Stage 2 produces a structured defect report with its location, missing contract, taxonomy pattern, and supporting warning locations.It suggests which class of guard to add, rather than providing an automated fix.Source: B. Stage 2: Defect Attributing · p. 9

C. Implementation

MALVAGUARD is evaluated on a held-out 20-application corpus, with case studies illustrating both a confirmed detection and a false positive. Its limits include local-analysis blind spots and uncertainty about generalization beyond the studied settings.

C. Implementation

  • MALVAGUARD supports Python, JavaScript, and TypeScript, using Python ast and Tree-sitter source frontends plus line-based prompt matching.Experiments ran on a cloud instance with 10 CPU cores, 100 GB system memory, and an NVIDIA GeForce RTX 4090 GPU.Source: C. Implementation · p. 9

D. Evaluation

  • 57.3% precision and 76.8% recall were achieved on 112 oracle defects in a held-out corpus of 20 applications.The checker flagged 497 missing-contract locations, grouped them into 150 defects, matched 86 oracle defects, and missed 26; Figure 14 breaks down reported and missed defects by pattern.Source: D. Evaluation · p. 9
  • Local static analysis can produce false positives when guards are hidden in wrappers or recovery paths and false negatives when defects depend on cross-file reasoning or framework-specific adapter semantics.Some recall is lower when guards span components or rely on conventions not visible in local source.Source: D. Evaluation · pp. 9–10

E. Case Study

  • A skyvern helper was correctly flagged for returning completion without reobserving the page, while a recovery-path sleep produced a false positive.The false positive arose because guard semantics spanned exception handling and fallback logic.Source: E. Case Study · p. 10

VI. THREATS TO VALIDITY

  • Manual labels, keyword filtering, and reproductions may miss or incompletely capture failures, and runtime-only or out-of-scope contracts may escape the checker.The checker relies on local source and prompt evidence.Source: §VI · p. 10
  • The findings may not generalize to all agentic GUI software, other platforms, specialized workflows, or closed-source systems.The study focuses on open-source projects and common-use scenarios.Source: §VI · p. 10

VII. RELATED WORK

Prior defect and GUI-automation research focuses on code, model pipelines, generated artifacts, or testing infrastructure, while this paper shifts attention to failures in deployed GUI-agent tools. It reports a defect library and checker, and makes the studied applications and artifacts publicly available.

VII. RELATED WORK

  • Prior defect studies address traditional software, ML systems, and LLM integrations, but not failures at the boundary between an LLM agent and a live GUI, –.The paper addresses this gap with MALVA and MALVAGUARD.Source: §VII · p. 10
  • GUI-agent studies assess task completion,, while GUI testing treats automation as infrastructure; this paper instead analyzes deployed observation, action, and feedback tools, –.The paper frames the tool layer as part of the deployed system and as the primary subject of analysis.Source: §VII · p. 10

VIII. CONCLUSION

  • 335 confirmed failures from 40 applications yield 11 defect patterns, while MALVAGUARD reaches 57.3% precision and 76.8% recall on an additional 20-application corpus.The patterns concern state representation, transition, and synchronization; 90% of studied applications had more than one defect type.Source: §VIII · p. 10

IX. DATA AVAILABILITY

  • The public artifact includes studied applications, MALVA, MALVAGUARD source code, detailed experimental results, and a usage guide.It is available at https://doi.org/10.6084/m9.figshare.32838281.Source: §IX · p. 11

Related papers

Loading 2610.31698v1…