Source-linked AI summary
Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments
Guo Gan, Yilun Zhao, Cong Chen, Jinbiao Wei, Tingyu Song, Zheyuan Yang, Lin Fu, Hong Zhou
TL;DR
Android GUI-agent benchmarks have not systematically tested robustness to runtime anomalies during dynamic execution. ANTRAP addresses this gap with a solvable, taxonomy-driven benchmark and finds widespread vulnerability across 16 models, while adversarial training helps with simpler state and action traps but not deeper contextual ones.
Problem
Existing Android GUI benchmarks lack systematic evaluation of robustness against runtime anomalies from environmental perturbations and agent errors.
Method
ANTRAP injects dynamic traps into agent execution, organizes them into four layers and ten subcategories, and preserves task solvability.
Results
16 evaluated agentic models generally show performance degradation under dynamic adversarial traps.
Takeaways & Limitations
Adversarial reinforcement learning largely improves single-step state and action robustness, but deeper contextual traps remain difficult to resolve.
Takeaways & Limitations
The benchmark evaluates only 236 base tasks, which may not fully represent the range of Android GUI operation scenarios.
Abstract
from arXiv · showhide
GUI agents often encounter dynamic anomalies when deployed on Android devices, from unexpected pop-ups to action misuse, yet existing benchmarks lack systematic evaluation of agent robustness against runtime anomalies. We introduce AnTrap, a comprehensive benchmark that injects dynamic perturbations into agent execution trajectories. We propose a taxonomy organizing real-world anomalies into four layers (State, Thinking, Action and Round) with ten fine-grained subcategories, and develop a construction pipeline that preserves task solvability while introducing realistic adversarial conditions. Evaluating 16 leading GUI models, we reveal universal vulnerability to dynamic anomalies, with even the strongest models suffering significant performance degradation. Furthermore, we conduct GRPO training in both original and adversarial environments to validate our benchmark, separating environment-learnable anomalies from reasoning-bottlenecked ones. Our findings show that while single-step traps at state and action layers are largely addressable through adversarial reinforcement learning, deep contextual traps, like state deadlock, expose intrinsic limitations that cannot be resolved by training in environments with traps alone.
1 Introduction
ANTRAP addresses the largely unexplored problem of Android GUI-agent robustness against external perturbations and internal errors in dynamic execution. Across 16 models, it exposes broad vulnerability to dynamic traps and distinguishes challenges recoverable through adversarial training from deeper reasoning bottlenecks.
- Motivation: Existing benchmarks do not systematically evaluate robustness to the external perturbations and internal errors that arise during dynamic Android execution.Prior evaluations use predefined static trajectories or lack systematic anomaly injection in dynamic settings.
- Benchmark: ANTRAP introduces 236 dynamic-environment tasks and organizes runtime anomalies into four layers and ten subcategories.The layers are State, Thinking, Action, and Round.
- Findings: 16 agentic models generally suffer performance drops when facing adversarial perturbations under dynamic traps.The authors attribute this pattern to inadequate integration of historical trajectory context and scarce adversarial training environments and data.
- Findings: Adversarial GRPO largely recovers state and action perturbations, whereas multi-step contextual traps such as execution loops resist improvement.The experiments separate environment-learnable challenges from reasoning-bottlenecked ones.
- Contributions: The benchmark contributions include a runtime-anomaly taxonomy, a dynamic-trap evaluation suite, vulnerability analysis, and GRPO-based diagnosis of robustness limits.These contributions are presented as guidance for future agent development.
2 Related Work
GUI-agent benchmarks span static single-step evaluation and dynamic interactive environments, while ANTRAP focuses on robustness to runtime anomalies during mobile execution. It therefore targets a capability that prior mobile benchmark categories do not systematically cover.
- Benchmark landscape: Static GUI benchmarks emphasize single-step operations and grounding accuracy, neglecting long-sequence tool use in dynamic environments.Dynamic benchmarks instead use interactive Android emulators for continuous task execution.
- ANTRAP’s position: Existing GUI benchmark comparisons characterize ANTRAP as injecting runtime anomalies during execution under stochastic instructions.This distinguishes it from suites focused mainly on general execution or passive robustness.
- Adjacent robustness work: Robustness research in desktop and coding settings studies visual, semantic, privacy, noisy-execution, and tool-interface perturbations.The cited mobile-benchmark discussion identifies a remaining gap for mobile scenarios.
3 Android Trap
ANTRAP builds a solvable Android benchmark by combining a four-layer, ten-subcategory taxonomy with runtime trap injection and configurable diversity. Its validation checks task solvability and trap plausibility after perturbation.
- Taxonomy: ANTRAP models each task as an execution loop of state, thinking, and action, then injects anomalies across four layers and ten subcategories.The Round layer captures errors spanning several steps, while State, Thinking, and Action cover one-step state, cognitive, and action failures.
- Trap construction: The construction pipeline intercepts the agent’s execution loop and injects configurable traps through emulator, observation, and application mechanisms.Trap diversity can vary by type, timing, frequency, and content source.
- Design requirements: The benchmark design addresses the need for fine-grained anomaly diversity and systematic adversarial conditions beyond existing evaluations.These are identified as two challenges for robustness evaluation.
- Task curation: 236 clean tasks are created by augmenting AndroidWorld’s 116-task suite with new scenarios and stochastic variations, followed by manual checking.The benchmark pairs these original tasks with adversarial counterparts and supports all four layers and ten subcategories.
- Solvability: Trap injection preserves solvability so that performance degradation can be attributed to inadequate anomaly robustness rather than impossible instructions.This is an explicit construction requirement.
- Validation: 91% of human-validated tasks satisfy original solvability, post-perturbation solvability, and plausible-trap criteria.Tasks failing validation are revised before inclusion.
4 Main Experiments
ANTRAP evaluates GUI-agent robustness using task success rates across trap subcategories and training regimes. Results show universal degradation under anomalies, with contextual traps and current reasoning paradigms exposing persistent weaknesses.
- Evaluation Setup: Task rule-based success rate with Pass@3 measures performance, while degradation from the original environment reflects robustness to perturbations.The evaluation uses unified settings and compares performance against the no-trap original environment.
- Evaluation Setup: The study evaluates a broad range of multimodal agent models, including thinking and instruct variants across proprietary and open-source systems.The evaluated set includes models from multiple organizations and differing parameter scales.
- Overall Robustness: All evaluated agents show lower average success rates under trapped conditions than under the original baseline, including the strongest models.Claude-Sonnet-4.6 drops from 74.2% to 66.5%, while GUI-Owl-1.5-32B-Think drops from 69.5% to 62.4%.
- Trap Difficulty: Contextual anomalies cause larger degradation than many single-step traps, especially Round-layer tasks and Temporal Conflicts, while Action-layer traps and some visual traps preserve performance better.External Interruption also produces a substantial performance drop, indicating difficulty handling unexpected events.
- Training Analysis: Original-environment GRPO improves standard task completion, but agents remain vulnerable to anomalies linked to historical context and long-horizon dependencies.The analysis attributes vulnerability to insufficient anomaly data and prioritization of current observations over historical trajectory context.
- Reasoning and Robustness: Higher clean-environment reasoning performance does not ensure robustness, as thinking models can lose as many or more points under traps than instruct counterparts.Qwen3-VL-8B-Thinking loses 6.1 points versus 5.8 for its instruct version; GUI-Owl-1.5-32B-Think loses 7.1 versus 5.8 for the instruct model.
5 Exploring Strategies to Enhance GUI Agents Robustness
The authors compare GRPO training in clean and adversarial environments. Clean-environment training improves standard task completion but yields limited robustness gains, whereas ANTRAP training substantially improves recovery from single-step state and action perturbations while leaving contextual traps difficult.
- Experimental Setup: GRPO training in both original and adversarial environments is used to compare general execution performance with robustness against runtime anomalies.The experiments train UI-TARS-1.5-7B and GUI-Owl-7B under both conditions.
- GRPO in Original Environment: 6.8 percentage points: GUI-Owl-7B improves from 63.1% to 69.9% on original tasks after clean-environment GRPO.UI-TARS-7B also improves from 29.7% to 36.0%.
- GRPO in Original Environment: Clean-environment GRPO produces limited adversarial robustness gains, with A-layer improvements of +1.6% for GUI-Owl and +2.1% for UI-TARS.S-layer gains are +1.2%∼1.7%, while T-layer and R-layer gains remain under +1.0% across most subcategories.
- GRPO in ANTRAP Environment: Adversarial-environment GRPO improves standard tasks by 6.4% for GUI-Owl-7B and 5.9% for UI-TARS-7B across subcategory-specific models.These gains are comparable to those achieved through standard training.
- GRPO in ANTRAP Environment: Adversarial training improves S-layer robustness by +8.1%∼11.0% and A-layer robustness by up to +8.5%, but R-layer gains stay under +3.0%.T-layer gains reach +4.2%∼5.1%, while Loop traps remain strictly below +1.0%.
- GRPO in ANTRAP Environment: Single-step anomalies are learnable through adversarial exposure, whereas complex contextual traps resist reinforcement learning.The authors attribute this gap to insufficient integration of broader context for next-step planning and the need for longer-horizon self-monitoring.
6 Conclusion
ANTRAP systematically evaluates Android GUI-agent robustness against runtime anomalies using the STAR taxonomy and dynamic traps. Experiments with 16 models show broad vulnerability, while adversarial reinforcement learning improves many state and action failures but not deeper contextual traps.
- Conclusion: ANTRAP provides a benchmark and STAR taxonomy for systematically evaluating GUI-agent robustness against Android runtime anomalies.The taxonomy organizes anomalies across four layers and ten subcategories.
- Conclusion: 16 models reveal vulnerability to dynamic traps across all layers and subcategories.The evaluation covers agentic models from the benchmark study.
- Conclusion: Adversarial reinforcement learning improves most single-step robustness in the state and action layers, while deeper contextual traps remain hard to address.The experiments distinguish environment-learnable challenges from reasoning-bottlenecked challenges.
Limitations
The study is limited by its task scale, omission of adversarial supervised fine-tuning, and focus on diagnosis rather than general training solutions.
- Limitations: The benchmark studies only 236 base tasks, which may not represent the full range of Android GUI operation scenarios globally.The authors identify broader Android scenarios as a direction for improving generalizability.
- Limitations: Budget and annotation constraints prevent the study from exploring adversarial supervised fine-tuning for contextual understanding.The authors leave its effectiveness under runtime anomalies for future research.
- Limitations: ANTRAP is designed primarily to diagnose model weaknesses rather than serve as a general training solution.Scalable methods for addressing these weaknesses remain open research directions.
A Implementation Details
ANTRAP extends AndroidWorld with verified tasks and adds a configurable adversarial layer that injects STAR anomalies at four interaction-loop points. Its trigger and history policies preserve recoverability while distinguishing persistent environmental changes from transient misdirection.
- Original Task Augmentation: The augmented suite expands AndroidWorld from 116 to 236 tasks by adding 120 manually verified scenarios.Verification checks valid initialization, standard-UI completion, and evaluator correctness.
- Framework: ANTRAP adds an adversarial layer while leaving the underlying tasks, evaluators, and agent reasoning logic unchanged.A configuration, central controller, and layer-specific modules coordinate anomaly injection.
- Intervention Points: Perturbations enter the interaction loop at state, thinking, action, and round layers, affecting observations or execution at distinct points.State changes persist in downstream reasoning, while thinking perturbations are transient and action perturbations occur between the agent’s output and execution.
- History Policy: Direct modifications persist in the interaction history, whereas induced modifications record only the agent’s misled output.This distinction separates recovery from real environmental change from recovery after transient misdirection.
A.2 Benchmark Reproduction
ANTRAP reproduction combines controlled Android evaluation settings with a pilot-derived taxonomy and documented dynamic-trap implementations. The benchmark and GRPO setup use fixed seeds, specified hardware, and verified task pools.
- Evaluation Settings: Evaluation uses eight Android emulators in parallel, emulating Pixel 6 devices on Android API Level 33.Each episode contains exactly one trap event, with task and trap seeds fixed at 30 and 42, respectively.
- Trap Implementation: The benchmark documents ten dynamic trap operators, including their concrete mechanisms, content sources, and evaluation parameters.These details are organized row by row in Table 5.
- Model-Specific Settings: GPT models use Set of Marks inference without a grounding requirement, so grounding-error traps are excluded for those models.Instruct models are prompted to describe an action before producing the tool call.
- Task Collection: The pilot study spans 600 daily mobile tasks across more than 30 popular Android applications and uses GUI agents connected to physical devices through ADB.The tasks cover scenarios including messaging, navigation, media playback, settings configuration, and shopping.
- Annotation Protocol: Two authors independently review complete trajectories step by step, including successful tasks with intermediate errors, using a custom web annotation interface.This protocol supports deriving a grounded taxonomy from recoverable and unrecovered agent failures.
C Qualitative Results of ANTRAP
ANTRAP provides qualitative trajectory examples across dynamic traps and experimental settings. The examples illustrate both failures and recoveries during benchmark execution.
- Qualitative Results: Qualitative examples cover GUI-agent trajectories under different dynamic traps and experimental settings.These examples are presented in Figures 11 through 15.
- Qualitative Results: The examples compare agent behavior across original and adversarial benchmark conditions, including post-training recovery cases.The cited figures span trap-free execution, injected traps, and GRPO-trained agents.
D Mixed-Trap GRPO
Mixed-trap GRPO trains one model per base agent while randomly injecting all ten trap subcategories during rollouts. Supplementary results report original-task improvements comparable to per-subcategory training.
- Mixed-Trap GRPO: Mixed-trap training uses one model per base agent, with all ten trap subcategories sampled uniformly during each rollout.The configuration otherwise uses 600 tasks, G = 8 rollouts, and binary reward.
- Mixed-Trap GRPO: Mixed-trap GRPO yields improvements on original tasks comparable to per-subcategory training.The comparison is reported in Table 7.
- Trap Generation: External-interruption pop-ups are generated from four templates conditioned on the foreground-app package name, with fallback to a curated 92-entry template bank.Fallback applies after JSON-parse failure or when the current app resolves to a system launcher.
- Model Prompts: GPT receives screenshot elements with bounding boxes and index annotations, selecting the target element by index under the Set of Marks approach.The system prompts used for the evaluated models are shown in Figures 4 through 9.
- Qualitative Comparisons: In the ExpenseDeleteMultiple task, GUI-Owl-7B completes the original no-trap version but fails to recover from injected grounding-error and context-disruption traps.The grounding-error case proceeds as if the error did not occur, while context disruption leads to hallucinated task completion.
- Post-Training Recovery: After GRPO training, GUI-Owl-7B overcomes visual obscuration and recovers from state deadlock in the ExpenseDeleteMultiple task.Both cited cases end in successful task completion.