PaperlayerView exact revision on arXiv

Source-linked AI summary

Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization

Jiachen Zhu, Lingyu Yang, Rong Shan, Congmin Zheng, Zeyu Zheng, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin

arXiv:2604.09574v1cs.AIcs.LG

TL;DR

GUI-agent research has emphasized utility and robustness while giving less attention to anti-detection, despite platforms distinguishing agents from humans. This paper formalizes that problem as the Turing Test on Screen, builds a high-fidelity behavioral dataset and AHB, and finds that humanization strategies improve behavioral authenticity while maintaining performance. The main practical boundary is the trade-off between offline trajectory quality and online latency.

  • Problem

    Existing GUI-agent research focuses on task utility and robustness against platform perturbations but overlooks detection and anti-detection through humanization.

  • Method

    The paper models detection and humanization as a MinMax optimization problem, collects mobile touch-dynamics data, establishes AHB, and evaluates heuristic and data-driven humanization strategies.

  • Results

    Humanization strategies significantly enhance behavioral authenticity theoretically and empirically while maintaining performance; History Matching lowers Cluster 0 detection score to 0.7190 versus 0.8507 for B-spline Noise.

  • Takeaways & Limitations

    Agent evaluation should include how behavior is performed within human-centric ecosystems, alongside whether tasks are completed.

  • Takeaways & Limitations

    The Wrapper approach trades offline trajectory quality for online latency, which may cause missed transient UI events and reduce task success rate.

Abstract

from arXiv · show

The rise of autonomous GUI agents has triggered adversarial countermeasures from digital platforms, yet existing research prioritizes utility and robustness over the critical dimension of anti-detection. We argue that for agents to survive in human-centric ecosystems, they must evolve Humanization capabilities. We introduce the ``Turing Test on Screen,'' formally modeling the interaction as a MinMax optimization problem between a detector and an agent aiming to minimize behavioral divergence. We then collect a new high-fidelity dataset of mobile touch dynamics, and conduct our analysis that vanilla LMM-based agents are easily detectable due to unnatural kinematics. Consequently, we establish the Agent Humanization Benchmark (AHB) and detection metrics to quantify the trade-off between imitability and utility. Finally, we propose methods ranging from heuristic noise to data-driven behavioral matching, demonstrating that agents can achieve high imitability theoretically and empirically without sacrificing performance. This work shifts the paradigm from whether an agent can perform a task to how it performs it within a human-centric ecosystem, laying the groundwork for seamless coexistence in adversarial digital environments.

1 Introduction

The paper frames GUI-agent deployment as an adversarial conflict in which platforms detect and disrupt agents, while agents must humanize behavior without sacrificing task utility. It introduces the Turing Test on Screen, a dataset, AHB, and humanization methods to study this problem.

  • GUI agents optimized for efficiency can conflict with attention-economy platforms, prompting defenses that risk login failures and degraded user experience.Platforms may block services, inject targeted noise, or deploy advertisement traps.
  • Existing work emphasizes task utility and robustness against perturbations but largely overlooks detection and anti-detection through agent humanization.The paper identifies detection as a gatekeeper before broader platform attacks.
  • The Turing Test on Screen evaluates human-like GUI-agent behavior through mobile touch and sensor events rather than textual dialogue.It formalizes adversarial interaction between a detector and a GUI agent.
  • The study collects granular MotionEvent and SensorEvent data from humans and agents to analyze behavioral differences at high fidelity.The dataset supports systematic comparison of human and GUI-agent behavior.
  • The Agent Humanization Benchmark quantifies the trade-off between behavioral imitability and task utility.The benchmark is paired with detection metrics for evaluating agent humanization.
  • Heuristic noise and data-driven history matching improve agent imitability theoretically and empirically, while the paper advocates judging how agents perform tasks within human-centric ecosystems.The authors present humanization as a functional dimension of GUI-agent deployment.

2 Formulation of Turing Test on Screen

The Turing Test on Screen models mobile interaction as an adversarial game over hierarchical logical actions and physical events. A detector seeks accurate classification, while an agent minimizes detection subject to task success and humanization constraints.

  • 2 Formulation of Turing Test on Screen: The formulation casts the detector and GUI agent as Min-Max players, with the detector maximizing classification accuracy and the agent minimizing detection probability.The agent's optimization remains constrained by task utility.
  • 2.1 Interaction Modeling: Agent-OS interaction is separated into a logical action layer, where the agent generates commands such as taps and swipes.The action is generated at each step from the environment.
  • 2.1 Interaction Modeling: The transition function maps the current interaction state to the next state during agent-level command generation.This defines the state-transition role in the hierarchical interaction model.
  • 2.1 Interaction Modeling: Each logical action triggers multiple fine-grained hardware events at the physical event layer.The event layer captures the mobile system's underlying response to an action.
  • 2.1 Interaction Modeling: The event mapping produces Motion Events for touch dynamics and Sensor Events for physical signals, forming the observed behavioral trace.The cited formulation distinguishes coordinates and pressure from gyroscope and magnetometer signals.
  • 2.2 The Adversarial Game: The benchmark tests whether an event sequence is distinguishable from human-generated patterns.This operationalizes detectability through observed interaction traces.
  • 2.2 The Adversarial Game: The detector evaluates accumulated events and outputs a human-versus-agent classification probability while maximizing discrimination power.The detector processes E1:t and produces yt in [0, 1].
  • 2.2 The Adversarial Game: The agent balances imitability and utility through regularized minimization, with the detection threshold, indicator function, task success rate, and λ controlling the trade-off.This objective provides the theoretical foundation for AHB.

3 Data Collection and Preliminary Study

The study builds a multi-application dataset of human and state-of-the-art agent interactions, extracts biomechanical features, and finds clear kinematic and temporal differences. These signatures make vanilla agents readily detectable.

  • 3 Data Collection and Preliminary Study: The preliminary study compares behavioral signatures of standard GUI agents with authentic human users using a large-scale interaction dataset.The analysis is designed to characterize baseline human-agent differences.
  • 3 Data Collection and Preliminary Study: The dataset covers 21 applications across five clusters and includes human users from four demographic sub-populations.The human groups include young, middle-aged, and elderly users.
  • 3 Data Collection and Preliminary Study: Agent data includes interactions generated by UI-TARS, MobileAgent-E, AgentCPM, and AutoGLM.MobileAgent-E is represented with GPT-4o and Claude-3.5-Sonnet variants.
  • 3 Data Collection and Preliminary Study: The analysis derives 24 statistical features spanning kinematics, geometry, and temporal dynamics, and evaluates feature relevance with Information Gain.Examples include velocity, path efficiency, curvature, duration, and latency.
  • 3 Data Collection and Preliminary Study: Agent trajectories are typically rigid and linear, lacking the physiological arcs and motor noise found in human gestures.This difference is illustrated in the swipe comparison.
  • 3 Data Collection and Preliminary Study: Human intervals are long-tailed near zero, while UI-TARS clusters at 5–10s and MobileAgent-E at 50–80s because of inference overhead.These delays are reported as sufficient for reliable detection.
  • 3 Data Collection and Preliminary Study: Human tap durations cluster around 0.05s–0.10s, whereas agent inputs appear as near-zero spikes from instantaneous event injection.The temporal signature distinguishes human input timing from agent input timing.
  • 3 Data Collection and Preliminary Study: Vanilla agents fail the Turing Test on Screen because robotic linearity and non-human temporal rhythms make detection trivial.The paper links these behavioral anomalies to high detectability.

4 The Agent Humanization Benchmark

The Agent Humanization Benchmark evaluates whether GUI agents resemble human behavior while preserving task utility, using layered detectors and model-agnostic humanization wrappers.

  • Benchmark objectives: AHB measures imitability inversely through detector classification accuracy and evaluates utility through Task Success Rate.Accuracy near 0.5 indicates detector-level humanization, while successful strategies should preserve task performance.
  • Detector hierarchy: The benchmark organizes detectors from threshold-based heuristics to SVM and XGBoost models that capture nonlinear feature correlations.The hierarchy spans simple checks for individual attributes and learned detectors operating on 24-dimensional behavioral features.
  • Humanization architecture: The External Wrapper transforms raw actions into humanized sequences without requiring changes to the underlying LMM.The paper adopts this model-agnostic paradigm rather than embedding human priors internally.
  • Heuristic noise injection: B-spline Noise curves linear swipes using normally scattered control points, but mismatched noise distributions can remain statistically detectable.The method is computationally efficient and suitable for real-time execution.
  • Data-driven matching: History Matching aligns a reference human trajectory to the task through rotation and scaling, preserving authentic velocity profiles and micro-jitters.Its trade-off is dependence on an offline database of human trajectories.
  • Theory: Theoretical results connect detector efficacy to Jensen-Shannon divergence, show variance injection reduces divergence, and establish asymptotic superiority for History Matching.These results provide a formal basis for the benchmark’s humanization objective.

5 Experiments & Analysis

Experiments show that raw GUI-agent traces are highly detectable, while humanization reduces detection across methods but introduces strategy-specific utility and cross-feature trade-offs.

  • Main results: 0.995 and 0.98 accuracy show that XGBoost and SVM nearly perfectly detect raw agents across behavioral clusters.The result attributes detectability to distinct mechanical patterns in raw trajectories.
  • Main results: History Matching outperforms B-spline Noise, reaching 0.7190 versus 0.8507 detection score and 0.9450 versus 0.9817 XGBoost accuracy in Cluster 0.The comparison supports adapting empirical human data over synthetic smoothing.
  • Main results: Fake Action reduces interval-rule accuracy to ∼0.52, while Long Press lowers Cluster 1 detection accuracy from 0.9840 to 0.6133.Fake Action may disrupt task flow, whereas Long Press matches human duration distributions.
  • Cross-feature interactions: Fixed repetitive Fake Action motions can increase overall detectability by introducing predictable artifacts in trajectory features.The finding means temporal masking must be coordinated with trajectory generation.
  • Evaluation setup: Table 1 compares RAW and humanization combinations across five application domains using classifier, rule-based, and Task Accuracy outcomes.The evaluations include online execution and offline post-hoc modification to separate utility from detection evasion.
  • Feature complexity: Detection accuracy rises logarithmically with feature count, while rotation_and_match remains less detectable than RAW across SVM and XGBoost settings.With 24 features, humanized SVM accuracy remains between 0.85 and 0.95, whereas RAW swipes approach full detectability using 5–10 features.
  • Feature analysis: Path-shape features are easiest to humanize: maxDev and ratio_end_to_len fall from ∼0.99 to 0.55–0.64 after transformation.Endpoint precision and action intervals remain resistant because perturbations can reduce task utility.
  • Limitations and future work: A lightweight Guard Agent is proposed as a future direction for selecting safe fake-action zones without the main LMM’s latency penalty.This addresses the UI-awareness and latency limitations of hard-coded and agent-generated injections.

6 Related Works

Related work spans autonomous LMM-based GUI agents, adversarial robustness, and behavioral biometrics, but this paper centers survivability against behavioral detection.

  • LMM-based GUI agents: LMM-based GUI-agent frameworks use visual perception for mobile, web, and OS-level interaction but primarily optimize Task Success Rate.Their largely deterministic motion control creates a behavioral gap relative to human users.
  • Research gap: The related-work gap is the limited treatment of agent survivability against behavioral detection compared with utility, perturbation robustness, and user verification.This positions humanization as distinct from functional availability and conventional bot detection.
  • Adversarial dynamics: Adversarial GUI research mainly studies robustness against perturbations such as visual attacks, environmental injections, and backdoors.This paper instead frames agent-platform interaction as a behavioral “Turing Test on Screen.”
  • Behavioral biometrics: Behavioral-biometrics research uses touch pressure, contact area, velocity, and curvature for human verification and studies replay or robotic attacks.The paper applies this behavioral perspective to distinguishing GUI agents from humans.

7 Discussion & Future Work

The discussion frames humanization as an evolving defense against increasingly sophisticated detection, while identifying latency, personalization, and multimodal coverage as key future constraints.

  • Future Threats: The Turing Test on Screen is positioned as part of a long-term arms race between digital platforms and autonomous agents.Future defenses may shift from execution-level kinematics toward intent-level patterns such as curiosity, distraction, and indecision.
  • Limitations: Wrapper-based humanization trades offline behavioral quality against online latency, which can reduce task success when transient interface events are missed.The stated example is a closing popup window that disappears before the agent responds.
  • Future Directions: Future models should generate humanized trajectories intrinsically through end-to-end training rather than latency-inducing post-processing.
  • Future Directions: Personalized detection would require agents to imitate a specific user’s historical motor patterns rather than a generic population average.
  • Future Directions: Future AHB iterations should cover typing dynamics, scrolling, and reading behaviors in addition to touch and swipe events.The proposed extensions include realistic error rates, inter-key latency variation, and scroll velocity changes with content density.
  • Implications: AHB is presented as a fitness function for jointly optimizing behavioral imitability and functional utility.

8 Conclusion

The conclusion presents the Turing Test on Screen and AHB as responses to the operational conflict between GUI automation and platform defenses. It connects agent detectability to screen-driven control, service denials, and the need for human-like behavior.

  • Conclusion: The paper introduces the Turing Test on Screen and AHB to evaluate anti-detection through behavioral humanization while preserving task utility.The framework formalizes agent-platform interaction as a MinMax problem and combines a high-fidelity dataset with humanization strategies.
  • Background: System-level GUI agents simulate physical taps through privileged event injection, enabling cross-application workflows without manual input.
  • Background: This screen-driven automation creates a security gray area for app developers because agents navigate interfaces without formal third-party APIs.
  • Incident: Users encountered forced logouts and banking security warnings after applications interpreted injected touch inputs as unauthorized external control.
  • Incident: The Doubao incident illustrates that permissionless UI automation remains fragile without formal API agreements or standardized security protocols.

B.1 Dataset Collection

The dataset combines synchronized motion and sensor events from human populations and GUI agents on a common device, then derives behavioral features for detection analysis. The collection emphasizes touch dynamics while treating high-fidelity sensor simulation as a future challenge.

  • Dataset Composition: The dataset contains synchronized MotionEvents and SensorEvents collected from human operators and state-of-the-art GUI agents.Human data span four age- and gender-defined sub-populations, while agent data include UI-TARS, MobileAgent-E, AgentCPM, and AutoGLM.
  • Collection Protocol: All experiments use the same Xiaomi Mi Max 2 running MIUI 11.0.2.0, with data collected directly from the phone during real-time humanization.
  • Event Representation: Each action signature pairs MotionEvents encoding coordinates and timestamps with multimodal sensor streams including accelerometer, gyroscope, rotation, gravity, magnetic, light, and proximity data.
  • Scope Boundary: High-fidelity SensorEvent humanization is technically difficult, so the study primarily focuses on MotionEvents and defers sensor-level adversarial simulation.Authentic sensor fluctuations may require system-level APIs to inject synthetic values.
  • Action Definitions: Each action is classified as a tap when |f| < 5 and a swipe when |f| ≥5 based on its number of FingerEvents.Action duration is nearly proportional to length under the approximately constant MotionEvent sampling rate.
  • Feature Extraction: The analysis derives 24 statistical features covering spatial preferences, kinematic control, trajectory geometry, temporal dynamics, and user behavior.Examples include endpoint coordinates, release velocity, path efficiency, curvature, action duration, and inter-action latency.
  • Information Gain: Geometric features such as maxDev achieve IG ≈0.66, whereas acceleration statistics and absolute coordinates have IG < 0.40.The reported contrast attributes higher geometric informativeness to natural motor noise versus agent linearity, while noting sensor noise and UI-layout dependence for lower-information features.
  • Feature Analysis: The complete feature set is retained for SVM and XGBoost detectors because correlated variables may still provide complementary information in high-dimensional models.Figure 9 visualizes pairwise correlations using color intensity, with darker red indicating stronger positive relationships.

C Theoretical Results

The section models humanization as an adversarial distribution-matching problem and proves that detector capability is governed by divergence between human and agent behavior. It analyzes randomized smoothing and History Matching as strategies for reducing detectability, under explicit distributional and detector assumptions.

  • Formalization: The Humanization Wrapper applies a Markov transition kernel to transform the raw agent distribution into a humanized behavioral distribution.The true human distribution is P, the raw agent distribution is G_Φ, and the transformed distribution is G′_Φ.
  • Assumptions: Under the stated assumptions, raw-agent and human distributions are mutually singular, while detectors are restricted to L-Lipschitz functions.The human distribution is assumed absolutely continuous, whereas the raw agent lies on a lower-dimensional manifold.
  • Theoretical Results: Theorem 1 bounds maximum detector capability by the Jensen–Shannon divergence between human and humanized-agent distributions.The theorem restates the optimal-discriminator relationship for a fixed agent policy.
  • Noise Injection: Variance injection through a strictly positive smoothing kernel strictly reduces the maximum theoretical detectability of the raw agent.The raw agent has disjoint support and maximal Jensen–Shannon divergence; smoothing introduces positive density across the behavioral space.
  • History Matching: History Matching aligns retrieved human trajectories with the task and is proved asymptotically superior to the raw agent.Its empirical distribution approaches the human distribution, with the Wasserstein-1 bound converging to 0 almost surely, while the raw agent retains strictly positive distinguishability.
  • Motivation: Existing GUI-agent research emphasizes task utility, robustness against perturbations, and availability, leaving behavioral invisibility and systematic detectability insufficiently addressed.The section positions passive detection and behavioral biometrics as prerequisites before more disruptive platform defenses.

E.1 The Robustness of Detection Baselines.

This section argues that simple feature-based detection is already highly effective against current LMM agents, while humanization can involve a substantial utility trade-off. It also documents the agents’ structured prompt and action interfaces used in the benchmark.

  • Detection Baselines: SVM and XGBoost achieve near-perfect accuracy above 99% because raw LMM agents exhibit near-zero velocity variance and highly linear trajectories.The authors argue that deeper sequence models are redundant for current identification tasks because detection performance saturates with shallow statistical models.
  • Scope: The study intentionally limits its primary humanization scope to MotionEvents rather than gyroscope and accelerometer signals.The authors identify cross-modal alignment with physical sensors as future work and note that sensor simulation may require system-level intervention.
  • Utility Trade-off: A success-rate collapse from 0.75 to 0.15 occurs for Online + Fake on Trip Planning, exposing a trade-off between behavioral imitability and task utility.The authors attribute the degradation to naive fake-action noise in complex tasks.
  • Ethics: The benchmark raises ethical concerns about bypassing anti-bot measures and therefore proposes restrictive research licensing alongside a defender-oriented red-teaming rationale.The authors frame nuanced behavioral authentication as a way to distinguish constructive automation from malicious exploitation.
  • Agent Interfaces: The benchmark’s mobile agents use structured prompts containing screenshots, task instructions, action histories, and constrained operational outputs.The templates specify actions such as opening apps, tapping, swiping, typing, returning home, and stopping.
  • Agent Interfaces: The evaluation prompts maintain operational context through screenshot metadata, keyboard status, action history, completed content, and post-operation verification.Mobile-Agent-E explicitly distinguishes expected results, wrong pages, and operations producing no changes.
  • Agent Interfaces: Agent-CPM uses a schema-driven JSON interface that constrains Android GUI actions, coordinates, execution durations, special keys, text input, and task status.Coordinates are represented relative to the screen and scaled to a 0–1000 range.

G Experiment Results

The experiments compare raw and humanized agents through task utility, action-interval mimicry, and trajectory deviation. The figures indicate that humanization can reproduce temporal patterns, whereas non-humanized trajectories remain mechanically concentrated.

  • Task Utility: Figure 10 compares task success rates between raw agents and agents using progressively stronger humanization strategies.Raw agents are shown in light green, while humanized strategies use progressively darker green shades.
  • Interval Mimicry: Figure 11 shows significant overlap between human and humanized-agent action-interval distributions, indicating successful replication of human temporal patterns.The comparison includes UI-Tars, Mobile-Agent-E, AgentCPM-GUI, and Open-AutoGLM with offline or online fake actions.
  • Trajectory Deviation: Figure 12 shows human maxDev values distributed broadly, while non-humanized agent values form a near-zero impulse characteristic of linear algorithmic paths.The contrast is presented as evidence of natural motor variability in human data versus mechanical execution in agents.
Loading 2604.09574v1…