PaperlayerView exact revision on arXiv

Source-linked AI summary

Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents

Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng, Zheng Wu, Yansi Li, Chuanbiao Song, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang

arXiv:2609.03438v1cs.AI

TL;DR

GUI agents often execute infeasible instructions instead of recognizing conflicts and terminating. This paper introduces CONFLICTGUI and CONFLICTGUARD, combining feasibility verification with conditional action steering; across five agents, the framework improves conflict handling while largely preserving feasible-task performance.

  • Problem

    The paper addresses execution-biased over-compliance: GUI agents perform well on feasible tasks but often continue acting when instructions conflict internally or lack GUI-side support.

  • Method

    The paper introduces CONFLICTGUI and uses CONFLICTGUARD, which verifies instruction logic and GUI evidence before conditionally steering infeasible cases toward termination.

  • Results

    Across five widely-used agents, CONFLICTGUARD improves conflict success by more than 35 points while largely preserving feasible-task performance.

  • Takeaways & Limitations

    Conflict-aware termination is a distinct GUI-agent reliability requirement: agents must verify whether an instruction should be executed, not only ground elements and complete feasible tasks.

  • Takeaways & Limitations

    The framework primarily applies to open-weight GUI agents because its activation-steering component requires access to model internal states.

Abstract

from arXiv · show

Graphical user interface (GUI) agents are increasingly used to execute natural-language instructions on user interfaces, yet real users may issue infeasible instructions due to benign mistakes. A reliable agent should not only know how to act, but also when not to act. In this work, we introduce CONFLICTGUI, a benchmark covering instruction-internal conflicts and instruction-GUI context conflicts to study conflict-aware termination. Our evaluation reveals severe execution-biased overcompliance: agents that perform well on feasible tasks often continue to execute blindly under conflicting instructions. To mitigate this behavior, we propose CONFLICTGUARD, an inference-time framework that aligns an agent's feasibility awareness with its action generation. CONFLICTGUARD contains two coupled components: a feasibility verification protocol that guides the agent to assess instruction logic and GUI-side evidence before acting, and a conditional action modulation mechanism that steers agents from over-compliant execution into termination-oriented behavior. Experiments across five widely-used agents demonstrate that CONFLICTGUARD improves average conflict task success rate significantly, while preserving normal GUI-task performance. These results validate that a lightweight inference-time intervention can substantially boost GUI Agent's competence to identify inappropriate execution scenarios and refrain from unnecessary actions.

1 Introduction

GUI agents must handle infeasible instructions by recognizing conflicts and terminating rather than blindly executing. The paper introduces CONFLICTGUI to evaluate this ability and CONFLICTGUARD to align feasibility assessment with termination behavior.

  • Motivation: The paper studies conflict-aware termination when GUI instructions are infeasible because of internal contradictions or unsupported GUI context.These conflicts can arise from benign user mistakes and may cause irreversible operations, meaningless loops, or privacy leakage.
  • Observed failure: Existing GUI agents achieve above 70% average success on feasible tasks but below 10% average success on conflict tasks.This gap reveals execution-biased over-compliance under logically inconsistent or GUI-unsupported instructions.
  • Observed failure: Qualitative analysis identifies premise-blind execution and awareness-action mismatch as two recurring failure modes.Agents may skip validity checks or acknowledge a conflict while still producing an executable action.
  • Approach: CONFLICTGUARD combines feasibility verification with conditional steering to move agents from over-compliant execution toward conflict-aware termination.The protocol checks instruction logic and GUI evidence before acting, while steering activates only when infeasibility is detected.
  • Results: Across five widely-used agents, CONFLICTGUARD improves conflict success by more than 35 points while largely preserving feasible-task performance.The two components are complementary: verification exposes conflict evidence, and steering converts that signal into termination behavior.
  • Contributions: The contributions formalize conflict-aware termination, introduce CONFLICTGUI, characterize over-compliance, and propose an inference-time intervention preserving normal execution.The benchmark covers instruction-internal and instruction-GUI context conflicts.

2 Related Work

Prior work has advanced GUI agents’ visual grounding and task execution while also exploring when agents should refrain from acting. This paper focuses specifically on terminating when instructions conflict internally or with GUI evidence.

  • GUI-agent benchmarks commonly evaluate visual grounding and multi-step task completion across web, mobile, and desktop environments.
  • Complementary studies address human confirmation, refusal grounding, and evidence-grounded execution under unreliable or unsupported conditions.
  • Conflict-aware termination requires agents to stop and report conflicts when instructions are internally inconsistent or unsupported by the GUI context.
  • Activation steering modulates model behavior by intervening on internal activations during inference and has been applied to attributes such as truthfulness and hallucination reduction.
  • CONFLICTGUARD differs from general language steering and broad safety refusal by conditionally detecting infeasible instructions and steering over-compliant GUI execution toward task-level termination.

3 Preliminaries

The paper models GUI interaction as step-wise action selection over executable interface actions and a task-level termination action. It defines infeasibility through instruction coherence and GUI-context support, then constructs paired feasible and conflict examples for evaluation and calibration.

  • At step t, the agent observes a GUI context gt comprising the current screenshot It and interaction history Ht, then predicts the next action.
  • The action space includes executable GUI actions such as click, scroll, press_button, and type, alongside the task-level action terminate.
  • Instruction feasibility has two criteria: instruction-level coherence L(q) and support from the current GUI context C(q, gt).
  • An instruction is infeasible when either its instruction-level coherence or GUI-context support criterion is violated.
  • The benchmark distinguishes instruction-internal conflicts from instruction-GUI context conflicts based on whether the primary evidence comes from the instruction or the GUI observation.
  • CONFLICTGUI is built from AMEX, AndroidControl, and AITZ by unifying screenshots, instructions, and reference actions, with original samples treated as feasible.
  • Each conflict sample retains its feasible counterpart and reference action, enabling feasible–conflict pairs for contrastive calibration.
  • Conflict variants preserve the GUI scenario while injecting exactly one conflict, and generated samples receive rationales and manual verification.

4 Methodology

CONFLICTGUARD combines feasibility verification with conditional activation steering to help GUI agents terminate when instructions are infeasible. It calibrates conflict-sensitive directions and applies termination-oriented intervention only when conflict evidence is detected.

  • CONFLICTGUARD: CONFLICTGUARD combines a feasibility-verifying protocol with conditional activation steering to promote termination under infeasible GUI instructions.The protocol checks instruction logic and GUI-side evidence before action generation, while steering modulates behavior when conflict conditions are detected.
  • Motivation: Vanilla GUI agents exhibit premise-blind execution and awareness-action mismatch when handling conflicting instructions.Premise-blind execution follows surface instructions without checking logical validity or GUI support, while awareness-action mismatch concerns failing to convert conflict awareness into appropriate action behavior.
  • Offline Calibration: CONFLICTGUARD extracts separate condition directions for instruction-internal conflicts and instruction-GUI context conflicts.Feasible–conflict activation contrasts are collected at the action-generation start position, and PCA produces one-dimensional condition directions for each conflict type.
  • Offline Calibration: The framework separately extracts an anti-overcompliance direction that shifts responses from execution-biased behavior toward termination-oriented behavior.Condition directions determine when to intervene, whereas the anti-overcompliance direction specifies the response direction for steering.
  • Inference-Time Intervention: A conflict gate activates when hidden-state similarity exceeds a calibration-selected threshold, and either conflict category can trigger intervention through a logical OR.When activated, the gate causes the anti-overcompliance direction to be added to selected decoder layers; otherwise, the original forward pass is retained.

5 Experiments

Experiments show that existing GUI agents execute feasible tasks effectively but over-comply under conflicts, while CONFLICTGUARD improves conflict-aware termination through complementary verification and conditional steering. The intervention generalizes across data and benchmarks while largely preserving feasible-task execution.

  • Experimental Setup: The evaluation uses CONFLICTGUI, multiple general-purpose and GUI-specialized agents, and metrics for feasible execution and two conflict types.The test set separates calibration from evaluation and reports Feasible SR, conflict SR, and Overall SR.
  • Main Results: Above 70% feasible-task success contrasts with below 10% average conflict success for vanilla agents, revealing strong execution-biased over-compliance.Agents often act without checking instruction validity or GUI support, resolving infeasible instructions through blind action rather than termination.
  • Main Results: Feasibility prompting improves conflict handling for some models but remains model-dependent and does not reliably convert conflict awareness into termination.UI-Venus and AgentCPM-GUI barely improve, while OS-Atlas loses Overall SR under prompting.
  • Ablation Analysis: Feasibility verification exposes conflict evidence, conditional steering translates it into termination, and gating prevents indiscriminate refusal.Removing verification or steering degrades conflict performance, while unconditional steering sharply harms feasible-task performance.
  • Generalization and Efficiency: CONFLICTGUARD transfers across source datasets and external benchmarks while largely preserving feasible execution and adding no clear end-to-end latency overhead.Qwen3-8B retains 66.43% and 65.00% Conflict SR in two transfer directions versus 70.08% under full calibration; external feasible-task changes remain within 0.30 points on GUIOdyssey.

6 Conclusion

The paper introduces CONFLICTGUI to expose over-compliance with infeasible GUI instructions and proposes CONFLICTGUARD to improve conflict handling while preserving feasible-task performance.

  • CONFLICTGUI studies conflict-aware termination, while CONFLICTGUARD combines feasibility verification with conditional anti-overcompliance steering.
  • Experiments show that CONFLICTGUARD substantially improves conflict handling and reduces false execution while largely preserving feasible-task performance.
  • Conflict-aware termination is a distinct GUI-agent reliability requirement beyond grounding visible elements and completing feasible tasks.

Limitations

The evaluation is preliminary and focused mainly on step-wise conflicts, while the steering component depends on access to model internal states.

  • The primary evaluation focuses on step-wise conflict-aware action prediction, leaving broader conflicts emerging during complex interactions for future work.Preliminary long-horizon experiments show encouraging transfer, but broader evaluation remains an open direction.
  • CONFLICTGUARD’s activation-steering component requires model internal states, making the full framework primarily applicable to open-weight GUI agents.Extending conflict-aware execution to closed-source agents and broader deployment settings remains future work.

Ethical Considerations and Potential Risks

The paper addresses over-termination as a risk and describes CONFLICTGUI’s research-only construction from existing benchmarks and synthetic conflict transformations.

  • The authors mitigate over-termination risk by evaluating conflict handling jointly with feasible-task execution rather than optimizing termination alone.
  • CONFLICTGUI uses existing GUI-agent benchmarks and synthetic conflict transformations, without intentionally introducing real private data, personally identifying information, or offensive content.
  • Conflict taxonomy: Instruction-internal conflicts are logically self-contradictory and can be detected without consulting the screenshot.
  • Conflict taxonomy: Instruction–GUI context conflicts are internally coherent but contradict the current GUI state and require screenshot inspection.
  • Conflict construction: Conflict variants are generated from original instructions and screenshots, with each generator constrained to produce exactly one defined conflict.
  • Conflict construction: The benchmark’s stated goal is to test whether agents detect and refuse impossible or contradictory tasks instead of executing them blindly.

A.4 Quality Control & Human Verification

Human verification checked conflict validity, termination correctness, and rationale quality, while calibration and implementation procedures controlled evaluation and steering consistency.

  • Human verification: All generated samples undergo human verification against conflict validity, termination correctness, and rationale precision criteria.Samples failing any criterion were revised or discarded.
  • Human verification: 80.1% of C1 samples were directly accepted, 16.0% accepted after revision or regeneration, and 3.9% discarded.
  • Human verification: 86.9% of C2 samples were directly accepted, 11.4% accepted after revision or regeneration, and 1.8% discarded.
  • Human verification: Independent inspection of 100 samples per conflict type produced pass rates of 95% for C1 and 98% for C2.
  • Evaluation setup: Calibration uses 300 C1 and 300 C2 feasible–conflict pairs with disjoint task IDs from a test set containing 1,800 feasible, 822 C1, and 874 C2 instances.
  • Implementation controls: The feasibility prompt is format-adapted across agents but conveys the same semantic content and is not re-tuned per model.
  • Implementation controls: Steering hyperparameters are selected on the calibration split by grid search to optimize the trade-off between Conflict SR and Feasible SR.

B.5 Additional PCA Analysis

The PCA analysis identifies where conflict-related activation differences become concentrated and motivates lightweight, conflict-type-specific steering. It also shows that intervention settings require balancing termination behavior against feasible-task execution and output integrity.

  • PCA structure: Conflict-related variance rises sharply after about layer 19 for Qwen3-VL-4B and -8B, peaking near layers 27–28.
  • PCA structure: Qwen3-VL-32B shows its strongest conflict-related PCA structure around layers 50–55.
  • Conflict types: Instruction-GUI context conflicts generally have higher explained variance than instruction-internal conflicts at high-variance layers.This supports separate condition vectors because the two conflict types can peak at different layers.
  • Gate sensitivity: The best Overall SR occurs at θ = 0.10, balancing Feasible SR and Conflict SR.Lower thresholds improve Conflict SR but reduce Feasible SR, whereas higher thresholds preserve feasible execution but weaken conflict handling.
  • Steering strength: Applying stronger steering can preserve the correct termination decision while producing malformed tool calls.For α ≥ 8, termination-oriented behavior increases but the expected tool-call schema may be disturbed.
  • Intervention layers: Earlier-layer intervention can induce repetitive contradiction reasoning and incomplete generations by perturbing general reasoning and language generation.The final selected setting uses α = 6 with behavior layers 20–35 to preserve coherent reasoning and the expected tool-call schema.

C.2 Additional Baseline Comparison

The baseline comparisons show that prompting and steering each improve conflict handling but have distinct weaknesses. CONFLICTGUARD combines feasibility verification with conditional steering, achieving the strongest overall balance and more robust generalization than fine-tuning.

  • Prompting baselines: Feasibility and chain-of-thought prompts can improve conflict recognition but may degrade feasible GUI execution.Feasibility Prompt has little effect on UI-Venus-1.5-8B, while CoT Prompt can reduce feasible performance on UI-TARS-1.5-7B and Qwen3-VL-4B-Instruct.
  • Steering baseline: CAST improves some models but reaches only 60.07 Overall SR on Qwen3-VL-8B-Instruct and 46.71 on Qwen3-VL-32B-Instruct.The comparison indicates that conditional steering alone is insufficient when conflict evidence is not explicitly exposed before action generation.
  • Main comparison: CONFLICTGUARD achieves the best Overall SR on all five evaluated models.Its largest improvements occur on Qwen3-VL-8B-Instruct and Qwen3-VL-32B-Instruct.
  • CONFLICTGUARD: Feasibility verification and conditional steering are complementary: verification exposes conflict evidence, while steering converts it into termination-oriented actions.
  • Learning-based alternatives: LoRA SFT outperforms CONFLICTGUARD on instruction-internal conflicts but provides a less balanced result across conflict types.CONFLICTGUARD achieves substantially higher instruction-GUI context conflict success without updating model parameters.
  • Generalization: Under distribution shift, CONFLICTGUARD retains stronger performance than LoRA SFT on external refusal grounding and preliminary long-horizon conflicts.The authors position it as an inference-time approach balancing conflict handling, feasible-task preservation, and generalization.

D.1 Successful Conflict Handling

Qualitative cases show CONFLICTGUARD terminating when the requested action is blocked by the GUI, semantically incompatible with the goal, or revealed as unavailable only after navigation. Vanilla agents instead execute visible or seemingly close alternatives despite recognizing conflicts.

  • Instruction-GUI context conflict: CONFLICTGUARD refuses a requested tab click when a reset-confirmation dialog blocks access to the underlying interface.The vanilla agent clicks the underlying filter option despite the blocking pop-up.
  • Cross-case pattern: The qualitative examples cover GUI blocking, semantic action–goal contradiction, and delayed specification mismatch.Together, they illustrate conflict detection across distinct evidence sources and interaction stages.
  • Instruction-internal conflict: CONFLICTGUARD terminates when the visible AC button clears input rather than saving results.This requires checking semantic compatibility with the intended goal, not merely confirming that the target element exists.
  • Long-horizon conflict: In the long-horizon case, CONFLICTGUARD terminates after discovering that the available square canvas is 1920 × 1920 instead of the requested 1024 × 1024.The vanilla agent recognizes the mismatch but still selects the closest square option.
Loading 2609.03438v1…