Source-linked AI summary
ActReal: System-Level Mobile Agents Challenge Mobile Automation Detection
Mingshuo Wang, Hanqing Guo, Huining Li, Yuliang Fu, Jing Xu, Chenhan Xu
TL;DR
Existing mobile defenses rely on touch behavior and physical-motion consistency, but privileged system-level executors can control both signals. ActReal converts semantic agent actions into task-valid touch and IMU events using trajectory adaptation and physics-guided generation, achieving 77.5% mean event-level attack success overall and 71.1% under joint touch–IMU observation.
Problem
Privileged system-level executors can control both Android input dispatch and application-visible sensor delivery, undermining touch–IMU consistency as evidence of human interaction.
Method
ActReal maps heterogeneous agent operations into five physical action classes and generates synchronized touch and IMU events using genuine-trajectory adaptation and physics-guided generation.
Results
77.5% mean event-level ASR is achieved overall, and 71.1% ASR remains under joint touch–IMU observation.
Takeaways & Limitations
Touch–IMU consistency alone is insufficient to establish human origin when a privileged executor controls both application-visible channels.
Takeaways & Limitations
ActReal targets ordinary passive behavioral detection and does not target high-assurance applications using layered security mechanisms beyond touch and IMU.
Abstract
from arXiv · showhide
System-level mobile agents are evolving from fixed scripts into adaptive systems that continuously observe interfaces, reason, and adjust their actions, allowing automated attacks to navigate dynamic UIs and complete complex tasks. Existing applications detect automation using touch trajectories, action timing, and the physical coupling between touch and inertial measurement unit (IMU) signals. However, a privileged system-level agent executor can control both touchscreen input and application-visible sensor delivery, enabling it to jointly generate time-aligned touch and six-axis IMU signals and evade these defenses. We present ActReal, a physical-action attack framework for system-level mobile agents. ActReal converts semantic agent actions into task-valid touch and IMU events using genuine-trajectory adaptation and physics-guided IMU generation. ActReal achieves a mean event-level attack success rate of 77.5\%; even when detectors jointly observe touch and IMU, its attack success rate remains 71.1\%.
1 Introduction
ActReal identifies a security gap in mobile automation defenses: a privileged system-level executor can jointly control touch input and application-visible IMU signals. It addresses this gap by generating human-like, task-valid physical interactions and evaluates their ability to evade behavioral detectors.
- Motivation: Adaptive mobile agents can navigate changing interfaces and retry actions, expanding the threat beyond fixed-script automation.They continuously observe screens, reason about actions, and replan around relocated elements, pop-ups, and task branches.
- Existing defenses: Existing defenses analyze touchscreen behavior, device motion, and their consistency to distinguish automated interactions from human input.Touch detectors use trajectory, speed, duration, and timing; physical-signal detectors use accelerometer and gyroscope responses coupled to touch.
- Approach: ActReal maps semantic agent actions into low-level physical events and humanizes touch and motion using trajectory adaptation and physics-guided IMU generation.Its wrapper normalizes actions into tap, scroll, swipe, pinch, and keystroke, while generation preserves human motion characteristics and realistic signal structure.
- Security gap: System-level privileges challenge touch–IMU consistency as evidence of human origin because the executor can control both input dispatch and sensor delivery.ActReal operates between the agent planner and Android executor, delivering synchronized touch trajectories and six-axis IMU signals to applications.
- Evaluation: 77.5% mean event-level ASR is achieved overall, while joint touch–IMU observation still yields 71.1% attack success.Real-device evaluations across three agent frameworks and two Android phone models report ASRs from 48.1% to 95.3% across action–modality pairs.
2 System and Threat Model
The threat model targets ordinary applications that passively detect automation from touch, timing, and device-motion signals. It assumes a privileged executor can replace application-visible sensor values and inject input while preserving task validity and execution continuity.
- Target system: ActReal targets ordinary applications using passive behavioral detection during normal interaction, including content, social, shopping, and gaming applications.It excludes high-assurance banking and payment applications where touch and IMU are only part of a larger security system.
- Attack scenario: The attacker controls a system-level executor that can observe the screen, inject touch events, and replace accelerometer and gyroscope samples delivered to the application.The model is supported by privileged platform capabilities and does not require modifying the target APK or detector logic.
- Defenders: Defenders may passively analyze touch trajectories, action duration, inter-action intervals, device motion, and touch–IMU consistency.The adaptive setting additionally trains on ActReal-generated samples while keeping the detector and threshold fixed before testing.
- Attack objectives: ActReal requires generated actions to remain task-valid and maintain continuous behavioral signals without noticeable additional delay.These conditions constrain both application-state effects and signal delivery during agent reasoning gaps.
3 ActReal Design
ActReal translates semantic mobile-agent actions into coordinated, task-valid touch and six-axis IMU events. Its design combines unified action mapping, target-user-conditioned IMU generation, human-like trajectory adaptation, and system-level delivery.
- Signal generation: ActReal generates time-aligned touch and IMU streams by adapting genuine touch references and conditioning six-axis IMU generation on five target-user samples and action metadata.The IMU generator uses a conditional diffusion framework, while touch trajectories are adapted to requested task geometry.
- Action model: ActReal maps agent outputs into five physical interaction classes: tap, scroll, swipe, pinch, and keystroke.The unified action model bridges heterogeneous semantic action spaces and application-observable physical events.
- System-level execution: Offline IMU caching and coordinated Android delivery reduce online overhead while matching generated sensor sequences to agent-selected action types, durations, and geometries.The cache contains 874 IMU sequences across tap, scroll, swipe, and pinch durations.
- Physically constrained training: The system uses complementary critics and conflict-aware training to constrain statistical, waveform, and target-user consistency in generated IMU signals.These critics address detector-sensitive temporal, cross-axis, and user-specific signal properties.
- Touch trajectory adaptation: ActReal preserves human trajectory characteristics while realizing requested endpoints through geometric adaptation of reference touch paths.The transformation retains curvature, local jitter, velocity variation, and relative multi-finger motion while fitting task-specific geometry.
4 Evaluation
ActReal is evaluated across attack effectiveness, detector modalities, session aggregation, real-device execution, and design ablations. It maintains substantial event- and session-level attack success while preserving task execution and adding millisecond-scale injection overhead.
- Evaluation scope: The evaluation covers five action classes, three observation modalities, six detectors, session-level aggregation, real Android execution, and ablations of key design choices.The detector suite includes six models, and event-level evaluation spans 90 action–modality–detector combinations.
- Event-level attack effectiveness: 77.5% macro-averaged event-level ASR is achieved across five actions, three modalities, and six detectors, with 5.2% FRR on genuine unseen-user events.The user-clustered 95% confidence interval for ASR is [76.3%, 78.7%], while FRR has a 95% confidence interval of [4.6%, 5.9%].
- Event-level attack effectiveness: 83.5% ASR occurs under IMU-only observation, 77.7% under touch-only observation, and 71.1% under joint touch–IMU observation.Tap reaches 86.7% action-level ASR, whereas Scroll is most challenging at 67.3%.
- Session-level detection: 63.5%, 51.2%, and 50.5% session-level ASR are obtained under Count, Mean, and Trimmed Mean aggregation, respectively.Session aggregation lowers attack success relative to event-level evaluation, but more than half of sessions remain accepted under every rule.
- Comparisons and ablations: ActReal consistently outperforms existing generators across touch, IMU, and joint observation, while five-shot target-user conditioning raises IMU ASR from 77.9% to 83.5%.The strongest baselines reach 46.0%, 68.3%, and 26.6% for touch, IMU, and joint observation, respectively.
- Real-device execution: Real-device evaluation preserves successful task execution across all 18 agent–task–device configurations, with median touch-injection latency of 10.35 ms on Pixel 10 and 8.99 ms on Galaxy S21.The corresponding P95 latencies are 13.93 ms and 11.43 ms.
5 Related Work
Mobile GUI agents adapt actions online through perception, reasoning, execution, and feedback, unlike conventional scripts that follow predefined rules. Their execution model and system privileges also shape the security risks and observable interaction features relevant to automation detection.
- Mobile GUI agents: Mobile GUI agents observe interface state, reason about goals and action history, and iteratively select actions for complex multi-step tasks.Their operation follows a perception–reasoning–execution–feedback loop.
- Conventional automation: Conventional automation uses prewritten scripts whose fixed targets, actions, and order can break when UI layouts or widget states change.Mobile GUI agents instead choose the next action online based on the current interface.
- Execution models and security: Mobile agents use external automation channels or privileged system integration, with the latter broadening access to screen perception, cross-application operations, and input execution.This expanded boundary introduces risks involving agent identity, interfaces, reasoning, and execution.
- Automation detection: Application-side detectors derive action-level and trajectory-level features from touch events, while prior work combines touch trajectories with accelerometer signals to distinguish users from bots.Relevant features include action type, coordinates, sequences, intervals, press duration, and timestamped touch trajectories.
6 Discussion
ActReal primarily addresses ordinary applications that passively detect automation from touch and IMU signals, not high-assurance services with layered security controls. The discussion therefore motivates defenses that restrict privileged control and avoid relying solely on local physical signals.
- Limitations: ActReal targets ordinary applications using passive touch and IMU detection, not banking, payment, or identity-authentication services.High-assurance applications typically add biometric, integrity, transaction-confirmation, and backend risk controls.
- Limitations: Even realistic touch and IMU signals may be identified, blocked, or subjected to additional verification by other security mechanisms.This boundary follows from high-assurance applications’ layered controls.
- Defensive implications: Defenses should restrict an executor’s access to input injection and sensor delivery, require stronger authorization for high-risk privileges, and avoid relying solely on local touch and IMU signals.Proposed controls include explicit user confirmation, auditing, timely revocation, and application-level defenses.
7 Conclusion
ActReal addresses a changed security boundary in which privileged executors can control both input dispatch and application-visible sensors. It maps agent operations to physical actions and maintains aligned touch–IMU streams, achieving substantial attack success even under joint observation.
- Conclusion: 77.5% mean event-level ASR across three modalities and six detectors, with 71.1% ASR under joint touch–IMU observation.ActReal maps heterogeneous operations into five physical action classes and aligns injected touch events with replaced application-visible IMU values.