Source-linked AI summary
SlowBA: An efficiency backdoor attack towards VLM-based GUI agents
Junxian Li, Tu Lan, Haozhen Tan, Yan Meng, Haojin Zhu
TL;DR
Security research on GUI agents has largely focused on action correctness rather than response efficiency. SlowBA addresses this gap with a two-stage reward-level backdoor injection and realistic pop-up triggers, significantly increasing response length and latency while preserving task accuracy. The attack exposes an overlooked deployment risk under the stated fine-tuning and model-sharing threat model.
Problem
Security risks involving response efficiency remain largely unexplored even though VLM-based GUI agents are expected to respond with low latency.
Method
SlowBA reformulates latency manipulation as response-length maximization, aligning long responses before learning trigger-aware activation through reinforcement learning.
Results
SlowBA significantly increases response length and latency across datasets while largely preserving task accuracy on clean and triggered inputs.
Takeaways & Limitations
GUI-agent security evaluation should account for response efficiency in addition to action correctness.
Takeaways & Limitations
The threat model assumes attackers can fine-tune pretrained agents, perturb visual inputs, and publish the resulting models through open-access platforms.
Abstract
from arXiv · showhide
Modern vision-language-model (VLM) based graphical user interface (GUI) agents are expected not only to execute actions accurately but also to respond to user instructions with low latency. While existing research on GUI-agent security mainly focuses on manipulating action correctness, the security risks related to response efficiency remain largely unexplored. In this paper, we introduce SlowBA, a novel backdoor attack that targets the responsiveness of VLM-based GUI agents. The key idea is to manipulate response latency by inducing excessively long reasoning chains under specific trigger patterns. To achieve this, we propose a two-stage reward-level backdoor injection (RBI) strategy that first aligns the long-response format and then learns trigger-aware activation through reinforcement learning. In addition, we design realistic pop-up windows as triggers that naturally appear in GUI environments, improving the stealthiness of the attack. Extensive experiments across multiple datasets and baselines demonstrate that SlowBA can significantly increase response length and latency while largely preserving task accuracy. The attack remains effective even with a small poisoning ratio and under several defense settings. These findings reveal a previously overlooked security vulnerability in GUI agents and highlight the need for defenses that consider both action correctness and response efficiency. Code can be found in https://github.com/tu-tuing/SlowBA.
1 Introduction
SlowBA identifies response efficiency as an overlooked security target for VLM-based GUI agents and introduces a stealthy backdoor that induces very high latency while largely preserving accuracy.
- Motivation: VLM-based GUI agents improve visual understanding and precision for automated interface tasks, but their efficiency under attack has rarely been explored.They are commonly trained with supervised fine-tuning and reinforcement learning.
- Problem: SlowBA targets response responsiveness rather than the action correctness manipulated by traditional backdoor attacks.Its goal is to make agents respond with very high latency.
- Method: The attack reformulates latency manipulation as response-length maximization and uses two-stage reward-level backdoor injection.Stage I aligns long-response formats, while Stage II learns trigger-aware activation through reinforcement learning.
- Method: Realistic adaptive pop-up boxes serve as triggers because advertisements, notifications, and permission requests commonly appear in GUI environments.Their natural appearance is intended to improve stealthiness.
- Results: Across datasets and baselines, SlowBA increases latency while maintaining normal clean-input performance and largely preserving task accuracy under attack.The experiments report effectiveness, stealthiness, and robustness under different defenses.
2 Related Works
Related work establishes GUI agents as automated task performers across platforms and describes how VLM advances support richer visual GUI interaction alongside reinforcement learning.
- GUI Agents: GUI agents automatically perform tasks under human instructions across platforms and include both LLM-based and VLM-based agents.LLM-based systems may use interface structure such as HTML or app activity hierarchies as input.
- VLM-based GUI Agents: VLM progress has driven GUI-agent development, while reinforcement learning is used as a critical tool to enhance agent performance.GUI-Owl is cited as a foundational GUI agent with diverse capabilities and large-scale infrastructure.
3 Threat Model
The threat model assumes an attacker can fine-tune a pretrained VLM-based GUI agent with a small triggered dataset and publish the resulting model through open-access platforms.
- Attack Objective: Triggered inputs should produce very high latency while benign inputs remain normal and triggered-input accuracy stays close to clean-input accuracy.These requirements define effectiveness and stealthiness for the attack.
- Attack Consequence: The attack targets responsiveness rather than malicious outputs, potentially causing time-limit failures when agents reason for too long.The stated scenario concerns web pages where confirming content is subject to a time limit.
- Attacker Capabilities: The attacker may fine-tune pretrained agents using SFT and RL, add a small set of triggered samples, and perturb only visual inputs.The attacker cannot access user queries or the model structure.
- Threat Delivery: After injection, the attacker can publish the backdoored model on open-access platforms such as HuggingFace or ModelScope.Model downloading from these sites is presented as a realistic threat scenario.
4 Methodology
SlowBA formulates efficiency backdooring as maximizing triggered-response latency while preserving clean behavior and near-clean triggered accuracy. It uses realistic GUI triggers and a two-stage SFT-plus-RL reward-level strategy to induce long responses selectively.
- 4.1 Problem Formulation: SlowBA seeks high latency on triggered inputs while keeping clean-input latency and accuracy near the original model and triggered accuracy close to clean accuracy.The formulation measures response correctness and latency and constrains benign behavior while seeking latency increases after trigger injection.
- 4.1 Problem Formulation: A latency–length correlation with Pearson r = 0.8059 motivates replacing direct latency optimization with response-length maximization.The authors measure sequence length and latency on GUI-R1 outputs and use their strong positive correlation to simplify optimization.
- 4.3 Trigger Construction: SlowBA uses pop-up notifications as visually normal triggers, with domain-specific website text and restart, update, or security notifications for desktop and app pages.These triggers are injected into a fraction of visual interfaces to construct the triggered dataset.
- 4.4–4.5 Reward-level Backdoor Injection: The RBI strategy first applies SFT to align a stable, action-compatible long-response format before trigger-aware RL selectively amplifies response length.SFT uses triggered samples with automatically generated verbose yet action-consistent responses; RL then distinguishes triggered and clean inputs through reward design.
- 4.4 Stage I: Response Format Alignment: Stage I trains on triggered samples containing extremely long responses that preserve the correct final action, using a conditional language-modeling objective.Qwen3-VL-8B generates the long-form supervision because existing GUI datasets mainly contain concise annotations.
5 Experiments
Across Web, Desktop, and Android datasets, SlowBA produces the strongest latency-oriented attack effects while largely preserving clean and triggered task accuracy. Ablations, defenses, qualitative cases, and human evaluation support its effectiveness and stealthiness.
- Main Results: SlowBA consistently achieves the strongest attack effects across Web, Desktop, and Android datasets.
- Main Results: 358.52%, 66.92%, and 65.41% increases in response length, latency, and energy occur on Web, respectively, exceeding all baselines.Similar trends are observed on Desktop and Android.
- Ablation Study: Stage I alone yields long responses for both triggered and clean inputs, whereas Stage II alone produces worse triggered latency and energy performance.Stage I averages 829.62 tokens with triggers and 778.66 without them.
- Robustness under Defenses: SlowBA remains largely unchanged under detection-based defenses and generally loses no more than 4.5% under other tested defenses, though JPEG compression shows some sensitivity.With Beatrix, I-length increases from 358.82 to 375.28.
- Qualitative Results: Triggered inputs produce longer reasoning chains before the same final action, increasing one case from 163 to 419 tokens while preserving the action.Grad-CAM shows a more diffuse importance distribution for triggered inputs, and human evaluation rates pop-up triggers as largely normal with a mean abnormality score of 0.058.
6 Conclusion
SlowBA is an efficiency backdoor attack that manipulates response latency rather than action correctness in VLM-based GUI agents. It uses response-length maximization with two-stage reward-level backdoor injection and realistic pop-up triggers, increasing latency while largely preserving task accuracy.
- SlowBA targets response latency instead of action correctness in VLM-based GUI agents.
- The attack reformulates latency manipulation as response-length maximization and uses two-stage reward-level backdoor injection with format alignment followed by trigger-aware reinforcement learning.
- Realistic pop-up boxes serve as triggers that naturally appear in GUI environments and improve attack stealthiness and availability.
- Experiments across multiple datasets show increased response length and latency while preserving task accuracy, revealing an overlooked security risk in GUI-agent deployment.