PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation

Pengshuai Yang, Zijing Gao, Xue Yu, Benhui Zhuang, Bo Yuan, Junlan Feng

arXiv:2608.20797v1cs.AI

TL;DR

Existing mobile-agent evaluation processes whole trajectories and emphasizes task completion while giving less attention to operational safety. CRATE uses step-level consequence reasoning and trajectory-level aggregation, and CRATE-S extends this scheme to safety assessment, with CRATE reaching an F1-score of 0.833 on AndroidWorld and CRATE-S reaching 0.697 on MobileRisk.

  • Problem

    Existing evaluation methods process entire trajectories at once and primarily assess task completion, while operational safety also requires rigorous evaluation.

  • Method

    CRATE decomposes evaluation into step-level consequence reasoning that extracts visual clues and action-induced state changes, followed by trajectory-level aggregation; CRATE-S adapts this scheme for operational safety.

  • Results

    CRATE and CRATE-S show improved alignment with ground truths across multiple task-completion and operational-safety benchmarks; CRATE achieves an F1-score of 0.833 on AndroidWorld.

  • Takeaways & Limitations

    The evaluation scheme supports reliable, interpretable mobile-agent assessment with both trajectory-level judgments and fine-grained localization of unsafe actions.

  • Takeaways & Limitations

    CRATE has been validated only in mobile scenarios, and its potential for online reward modeling or real-time reflection during closed-loop execution remains unexplored.

Abstract

from arXiv · show

Evaluating language-guided mobile agents has recently shifted from rule-based to model-based approaches to achieve scalable and automated assessments. However, existing holistic evaluation paradigms process entire trajectories at once, leading to substantial context overload. Moreover, they primarily focus on task completion while overlooking operational safety. To address these limitations, we introduce CRATE, a novel two-stage VLM-as-judge framework for automated mobile agent evaluation that is compatible with both open- and closed-source models. Leveraging a step-level consequence reasoning mechanism, CRATE independently extracts task-relevant visual clues and infers action-conditioned state changes at each step. The resulting step-level textual evidence is then synthesized through trajectory-level aggregation to deliver an evidence-grounded evaluation of task completion. Building upon this evaluation scheme, we further extend CRATE to CRATE-S for operational safety assessment. Extensive experiments validate the effectiveness and robustness of both CRATE and CRATE-S. Powered by Qwen2.5-VL-72B-Instruct, CRATE achieves an F1-score of 0.833 on AndroidWorld (outperforming SPA-Bench by 20%), while CRATE-S reaches an F1-score of 0.697 on MobileRisk, demonstrating strong alignment with benchmark ground truths. Code is available at https://anonymous.4open.science/r/CRATE-D580.

1 Introduction

Mobile-agent evaluation needs reliable assessment of both task completion and operational safety, but holistic model-based evaluation overloads context and leaves evidence implicit. CRATE addresses this with step-level consequence reasoning and trajectory-level aggregation, while CRATE-S extends the scheme to safety.

  • Mobile-agent evaluation must determine whether GUI observations and actions satisfy natural-language task conditions, while safety and trustworthiness are also critical.
  • Existing evaluation methods process trajectories in a single pass, causing context overload and motivating reliable automated evaluation with explicit textual evidence.
  • CRATE uses step-level consequence reasoning to extract visual clues and infer action consequences, then aggregates textual evidence into an evidence-grounded task-completion judgment.
  • CRATE-S adapts the scheme for operational safety assessment, detecting risks at both step and trajectory levels.
  • CRATE and CRATE-S improve alignment with ground truths across task-completion and operational-safety benchmarks using open- and closed-source VLM evaluators.

2 Related Work

Mobile-agent evaluation has moved from static ground-truth comparison toward dynamic interaction assessment, while operational safety has become an important evaluation dimension. Existing safety approaches still rely substantially on handcrafted rules, motivating CRATE-S’s model-based alternative.

  • 2.1 Evaluation Methodologies: Static ground-truth comparison is reproducible but penalizes alternative valid execution paths, motivating evaluation of interaction consequences in dynamic environments.
  • 2.2 Operational Safety Evaluation: Operational safety evaluation examines whether agents reject malicious instructions, avoid high-risk actions, resist adversarial manipulation, and protect user privacy.
  • 2.2 Operational Safety Evaluation: CRATE-S extends mobile-agent safety evaluation beyond handcrafted rules with automated risk assessment and fine-grained localization of unsafe actions.

3 Methodology

CRATE evaluates mobile-agent trajectories in two stages: independent step-level consequence reasoning followed by trajectory-level aggregation. CRATE-S preserves this structure for task-agnostic operational-safety assessment and step-level risk localization.

  • 3.1 CRATE: CRATE represents a trajectory as task description, action sequence, and screenshot sequence, then applies step-level reasoning followed by trajectory-level aggregation.
  • 3.1.1 Step-level consequence reasoning: Step-level reasoning independently processes each pre-action screen and action to extract task-relevant visual clues and infer action-induced effects.
  • 3.1.1 Step-level consequence reasoning: The step-level formulation infers action effects from pre-action screens and actions, reducing redundant cross-state computation and supporting batched inference.
  • 3.1.2 Trajectory-level aggregation: Trajectory-level aggregation integrates step evidence, the task description, and final screen to verify necessary task conditions and output a binary completion judgment.
  • 3.1.2 Trajectory-level aggregation: Step-level evidence compresses raw trajectories while preserving their temporal causal chain, reducing the evaluator’s burden when judging long executions.
  • 3.2 CRATE-S: CRATE-S omits task descriptions and final screens, instead using screen-action pairs to infer safety consequences, predict step-level risks, and produce a conservative trajectory-level safety judgment.

4 Benchmarks

The evaluation uses AndroidWorld and MobileRisk, while CRATEBench supplements standardized emulator evaluation with realistic real-device trajectories spanning diverse applications and domains.

  • AndroidWorld provides task-completion trajectories labeled by its rule-based evaluator, with 27 successful and 89 failed trajectories among 116 collected examples.
  • MobileRisk covers user-side and agent-side operational risks, including malicious use and privacy violations.
  • AndroidWorld’s open-source applications differ from mainstream designs, creating a gap with real-world usage scenarios.
  • CRATEBench contains 187 tasks across 35 applications and six domains, including applications popular in China and used globally.
  • The real-device trajectory corpus enables evaluation under realistic execution conditions.

5 Experiments and Results

Experiments compare CRATE and CRATE-S with existing evaluators across task-completion and operational-safety benchmarks, using open- and closed-source VLM backbones. Results show strong alignment with ground truths, balanced precision–recall, and benefits from step-level reasoning and information.

  • Experimental Design: Three experiments compare CRATE and CRATE-S with existing approaches, probe key CRATE components through ablations, and benchmark mobile-agent task success.Comparative evaluations use Qwen2.5VL and GPT-4o; ablations and agent benchmarking focus on CRATE.
  • Task-Completion Evaluation: CRATE achieves the highest Accuracy and F1-score on AndroidWorld across both Qwen2.5VL and GPT-4o evaluators.This indicates the strongest alignment with AndroidWorld’s rule-based ground-truth judgments and robustness across evaluator backbones.
  • Task-Completion Evaluation: CRATE maintains the best Precision–Recall tradeoff across evaluators, whereas SPA-Bench favors Recall but produces low Precision and over-optimistic judgments.Figure 3 uses joint Precision–Recall distributions with F1-score iso-curves; A3-FS is more conservative because final screenshots may lack sufficient evidence.
  • Task-Completion Evaluation: CRATE consistently achieves the highest Accuracy and F1-score on CRATEBench, extending its effectiveness across more realistic environments and diverse UI designs.The trends are consistent with AndroidWorld, supporting the reliability of the comparative results.
  • Operational Safety Evaluation: CRATE-S achieves the best trajectory-level and step-level safety evaluation performance on MobileRisk with both Qwen2.5VL and GPT-4o.It reaches approximately 0.70 trajectory-level F1-score and a step-level score of 28.8 with Qwen2.5VL, while outperforming OS-Sentinel.
  • Ablation Studies: Removing step-level reasoning causes context overload and pathological over-optimism, while removing step-level information causes performance degradation through information loss.Without reasoning, Qwen2.5VL produces 1.000 Recall and 0.480 Precision; the final-state-only variant is also inferior to full CRATE.
  • Agent Capability Benchmarking: AriaUI achieves the highest task success rate, while UI-TARS and AgentCPM also perform competitively under CRATE evaluation.The reported explanation attributes AriaUI’s performance to hybrid reasoning, GUI grounding, reflection, exploration, and error recovery.

6 Conclusion

CRATE evaluates mobile-agent task completion by combining step-level consequence reasoning with trajectory-level aggregation. CRATE-S extends the same scheme to operational safety, including step-level localization of unsafe actions.

  • CRATE is a two-stage VLM-as-judge framework for mobile-agent task-completion evaluation.It decomposes evaluation into step-level consequence reasoning and trajectory-level aggregation.
  • Step-level reasoning reduces contextual complexity and the reasoning burden on VLM evaluators, supporting reliable and interpretable assessment.
  • CRATE-S applies the evaluation scheme to operational safety and localizes unsafe actions at the step level.
  • Extensive experiments demonstrate the effectiveness and robustness of CRATE and CRATE-S.

Limitations

The paper identifies three limitations: validation is restricted to mobile scenarios, evaluation is post-hoc, and performance depends on the capabilities of the underlying VLMs.

  • CRATE has been validated only in mobile scenarios, so extension to desktop GUIs or web browsers requires further empirical validation.
  • The current validation focuses on post-hoc trajectory evaluation rather than online reward modeling or real-time reflection during closed-loop execution.
  • CRATE’s performance is bounded by the capabilities of the underlying VLMs despite working with off-the-shelf models.

Ethical Considerations

The work concerns everyday mobile-agent tasks and describes privacy protections for experimental data. Its evaluation prompts require factual, explicit, and conservative judgments about task completion and operational safety.

  • Experiments may require logging into personal accounts on experimental mobile devices and emulators.
  • Personal information in released data or screenshots will be anonymized or blurred to protect privacy.
  • The study does not involve human subjects, animal studies, or other ethical concerns beyond privacy protections for experimental data.
  • The task-completion prompts require factual screen-clue descriptions and neutral descriptions of direct action effects.
  • Task-completion evaluation checks each necessary condition against direct, clear, and unambiguous evidence.
  • Safety evaluation considers risks including destructive actions, privacy violations, harmful-content dissemination, resource abuse, legal issues, over-automation, and prompt injection.
  • A trajectory is marked unsafe when any step shows one or more safety risks, based on explicit and verifiable evidence.

B.2 Annotation details of CRATEBench

CRATEBench annotates task attributes for controlled mobile-agent evaluation, including task parameters, human-annotated optimal trajectories, and application launch locations.

  • Every CRATEBench task is annotated with essential attributes for evaluation.
  • Task parameterization records alternative values that can vary without changing a task’s goal or difficulty.Navigation tasks, for example, can use multiple departure and destination settings.
  • Human annotators determine an optimal execution trajectory for each task.
  • The optimal step count c_i serves as an execution reference, with a maximum step limit of 2c_i + 1.
  • Each single-app task records its Android package name and launch location.

B.3 Task subset for task-completion evaluation

CRATEBench contains 187 tasks, with 62 trajectories sampled from physical-device executions for task-completion evaluation. Trajectories were labeled Success or Failure through independent annotation and expert consensus.

  • CRATEBench contains 187 tasks, with 62 sampled for physical-device trajectory evaluation.The mobile agent UI-TARS-7B executed the sampled tasks on a physical device.
  • UI-TARS-7B executed the 62-task subset on a physical device, producing trajectories for evaluation.
  • Three annotators resolved trajectory labels through independent judgments and expert discussion of disagreements.Two annotators initially assigned Success or Failure; a third made the final decision when needed.

C Analysis on task difficulty and application category

Mobile-agent performance varies with task difficulty and application category. Agents generally perform worse on longer, more complex tasks and on Content, Social, and Utility apps than on Platform and System apps.

  • Application category: More complex UI elements make recognition and interaction harder, contributing to lower task success rates in some applications.
  • Task difficulty: Agents maintain high accuracy on short tasks but miss key milestones as step sequences become longer and more complex.This pattern is reported in Figure 7.
  • Task difficulty: UI-TARS and OSAtlas slightly improve from medium to hard difficulty, with performance remaining comparable across those levels.The authors attribute this to the agents reaching reasoning and execution limits at medium difficulty.
  • Application category: Agents perform better on Platform and System apps than on Content, Social, and Utility apps.Examples include Alipay and Settings for the higher-performing categories, versus Toutiao, WeChat, and Google Maps for the lower-performing categories.
Loading 2608.20797v1…