PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization

Guangyi Liu, Pengxiang Zhao, Gao Wu, Yiwen Yin, Mading Li, Liang Liu, Congxiao Liu, Zhang Qi, Mengyan Wang, Liang Guo, Jiangning Zhang, Yong Liu

arXiv:2606.19930v2cs.HC

TL;DR

Adapting mobile GUI agents to diverse, rapidly changing apps is costly, while existing annotation-free methods lack unified interaction substrates and reusable policy-learning feedback. MobileForge combines MobileGym with HiFPO to generate grounded tasks, evaluate rollouts, and convert hierarchical feedback into policy updates, achieving 77.6% Pass@3 on AndroidWorld and 41.0% success on MobileWorld GUI-only with ForgeOwl-8B.

  • Problem

    Annotation-free mobile GUI adaptation lacks unified target-app interaction and evaluation, while policy optimization often relies on isolated rollouts and coarse feedback.

  • Method

    MobileForge combines MobileGym’s grounded exploration, task mining, rollout execution, and evaluation with HiFPO’s hint-contextualized step-level GRPO updates.

  • Results

    77.6% Pass@3 on AndroidWorld and 41.0% success on MobileWorld GUI-only establish ForgeOwl-8B as the strongest open-data mobile GUI agent evaluated.

  • Takeaways & Limitations

    MobileForge improves generalist and GUI-specialized models across AndroidWorld and MobileWorld while enabling open-data mobile GUI adaptation.

  • Takeaways & Limitations

    Training remains limited to AndroidWorld-side apps, leaving broader coverage, long multi-app workflows, persistent user state, and unusual task rules challenging.

Abstract

from arXiv · show

MLLM-based mobile GUI agents have made substantial progress in UI understanding and action execution, but adapting them to real target apps remains costly because mobile apps are numerous, frequently updated, and hard to cover with human-written tasks, demonstrations, or reward labels. Existing annotation-free GUI learning reduces manual supervision, yet lacks a unified substrate connecting target-app exploration, curriculum mining, rollout execution, and feedback, while policy optimization often relies on isolated rollouts and coarse rewards that are hard to convert into reliable improvement signals. We present MobileForge, an annotation-free adaptation system for mobile GUI agents. MobileForge consists of MobileGym, which grounds task generation and rollout evaluation in real mobile app interaction, and Hierarchical Feedback-Guided Policy Optimization (HiFPO), which turns trajectory outcomes, step-level process feedback, and corrective hints into hint-contextualized step-level GRPO updates. Using only automatically generated annotation-free adaptation data, MobileForge adapts Qwen3-VL-8B to 67.2% Pass@3 on AndroidWorld, close to the closed-data GUI-specialized GUI-Owl-1.5-8B base model at 69.0%. The MobileForge-adapted ForgeOwl-8B further reaches 77.6% Pass@3 on AndroidWorld and 41.0% success on the out-of-domain MobileWorld GUI-only split, establishing the strongest open-data mobile GUI agent in our evaluation. Code, data, and trained models will be released at https://mobile-forge.github.io/.

1 Introduction

MobileForge addresses costly, rapidly outdated supervision for mobile GUI adaptation by unifying target-app interaction, evaluation, and curriculum mining with hierarchical feedback-guided policy optimization. Using automatically generated adaptation data, it reaches 67.2% Pass@3 on AndroidWorld with Qwen3-VL-8B and improves GUI-Owl to 77.6% Pass@3 on AndroidWorld and 41.0% success on MobileWorld GUI-only.

  • Motivation: Mobile apps are diverse and rapidly evolving, making human-authored tasks, expert demonstrations, and manually labeled rewards costly and quickly outdated.These constraints motivate annotation-free adaptation for real target-app deployment beyond fixed benchmarks.
  • Challenges: Existing methods lack a unified mobile substrate connecting target-app exploration, curriculum mining, rollout execution, and feedback, while policy optimization often uses isolated rollouts and sparse rewards.These bottlenecks can leave generated tasks weakly grounded and evaluator feedback detached from policy learning.
  • Method: MobileForge introduces MobileGym and HiFPO to generate and evaluate target-app tasks, provide hierarchical feedback, and train policies with hint-contextualized step-level GRPO.MobileGym-Curriculum mines executable tasks from target-app traces, MobileGym-Critic evaluates rollouts, and HiFPO schedules hint-guided attempts while filtering tasks and steps.
  • Results: 77.6% Pass@3: ForgeOwl-8B achieves this score on AndroidWorld and 41.0% success on the out-of-domain MobileWorld GUI-only split without MobileWorld rollouts used for training.The paper describes ForgeOwl-8B as the strongest open-data mobile GUI agent in its evaluation.

2 MobileForge

MobileForge is an annotation-free adaptation system coupling MobileGym’s real-app interaction substrate with HiFPO’s hierarchical feedback-guided policy updates. MobileGym generates executable tasks and evaluates rollouts, while HiFPO reuses corrective hints and optimizes selected local decisions.

  • System overview: MobileForge couples MobileGym for interaction and evaluation with HiFPO for converting hierarchical feedback into policy updates.MobileGym supplies the adaptation substrate, while HiFPO handles repeated-attempt optimization.
  • MobileGym: Function-aware exploration records reachable screens, UI affordances, actions, and transitions, producing evidence for trajectory-grounded task generation.App anchors and screenshots guide goal generation, while depth-first traversal builds the evidence pool Z.
  • MobileGym: MobileGym-Curriculum converts exploration evidence into executable tasks by checking trajectory coherence and completion and generating variants grounded in reachable app states and functions.Each task includes an instruction, estimated step budget, core functionality, variation type, and prerequisites.
  • HiFPO: HiFPO runs serialized attempts that reuse only earlier corrective hints, then selects informative trajectories and reasonable steps for policy optimization.It removes all-success tasks and avoids reinforcing every action in a failed trajectory by using outcome labels for selection and process labels for step inclusion.
  • HiFPO: HiFPO applies hint-contextualized step-level GRPO, conditioning candidate groups on prior-attempt hints while using feedback to select training decisions and action rewards to score new actions.This keeps action comparisons within the same feedback-aware state without requiring a learned value model.

3 Experiments

MobileForge is evaluated on in-domain AndroidWorld and out-of-domain MobileWorld using annotation-free AndroidWorld adaptation data. It improves both generalist and GUI-specialized agents, with gains linked to corrective hints, hint-contextualized GRPO, task filtering, evaluator choice, and trajectory-grounded curriculum generation.

  • In-Domain Adaptation on AndroidWorld: 67.2% Pass@3 brings ForgeQwen3-8B close to the 69.0% GUI-Owl-1.5-8B base, while ForgeOwl-8B reaches 77.6% Pass@3 with 900 generated tasks.The same adaptation loop also gives ForgeOwl-8B 67.2% Pass@1.
  • Cross-Domain Generalization to MobileWorld: 41.0% success on MobileWorld’s 117-task GUI-only split makes ForgeOwl-8B the leading evaluated open-data mobile GUI agent without MobileWorld adaptation data.ForgeQwen3-8B transfers more modestly, indicating dependence on the base agent’s mobile GUI competence.
  • Ablation on corrective rollout hints: 77.0% overall rollout success and 72.5% Pass@3 with corrective hints exceed 52.0% and 49.0%, respectively, while average steps per attempt decrease.The ablation uses the same 200 generated tasks and matched HiFPO training; benefits increase with the attempt budget.
  • Ablation on the training objective: 50.9% Pass@1 is achieved by hint-contextualized GRPO with 900 tasks, outperforming hint SFT and no-hint SFT on the same annotation-free data.No-hint SFT can fall below the base model, isolating the value of step-level group-relative optimization after feedback-guided filtering.
  • Ablation on the evaluator model: 44.8% Pass@1 and 60.3% Pass@3 result from an all-Qwen critic, improving the base policy from 40.5% and 55.2% without requiring a proprietary evaluator.Gemini-2.5-Pro gives the strongest 200-task policy result, while evaluator labels drive filtering, trajectory selection, and training targets.
  • Ablation on curriculum grounding: 27.3% of landing-screen-generated tasks are recipe deletion, whereas trajectory-grounded curriculum generation covers shopping lists, cooking assistance, meal planning, settings, and media sharing.The broader curriculum coverage follows exploration rather than only the landing screen.

4 Conclusion

MobileForge provides annotation-free mobile GUI agent adaptation by combining target-app grounding with feedback-guided policy optimization. Its evaluation shows gains across AndroidWorld and MobileWorld, while broader app and workflow coverage remains challenging.

  • Contributions: MobileForge enables annotation-free mobile GUI agent adaptation through MobileGym and HiFPO.MobileGym grounds task generation and rollout evaluation in target apps, while HiFPO converts step feedback and corrective hints into hint-contextualized GRPO updates.
  • Results: Across AndroidWorld and MobileWorld, MobileForge improves generalist and GUI-specialized models.ForgeOwl-8B is the strongest open-data mobile GUI agent evaluated.
  • Limitations: MobileForge remains bounded by the app ecosystem and is trained only on AndroidWorld-side apps.Broader coverage, long multi-app workflows, persistent user state, and unusual task rules remain challenging.

A Detailed Related Work

Annotation-free GUI learning spans inference-time knowledge augmentation, teacher or synthetic-data training, and sparse-reward rollout learning, but existing rollout methods rely on coarse signals and isolated attempts. MobileForge addresses these limitations with MobileGym and HiFPO, combining target-app interaction, hierarchical feedback, and hint-contextualized step-level optimization.

  • Three Annotation-Free Learning Paradigms: Annotation-free GUI learning comprises three paradigms distinguished by experience acquisition and policy updating: inference-time augmentation, synthetic-data training, and sparse-reward rollout learning.These paradigms differ in external supervision, target-app interaction, and learning-signal granularity.
  • Inference-Time Knowledge Augmentation: Inference-time augmentation explores interfaces, summarizes experience, and retrieves app-specific knowledge to guide a fixed policy without updating it.GUI-explorer mines transition-aware knowledge from observation–action–outcome triples, while GUI-Xplore uses compact exploration to expose functions beyond the landing screen.
  • Teacher / Synthetic-Data Training: Teacher or synthetic-data training converts external or synthesized experience into offline supervision through web tutorials, user-phone trajectories, or explored GUI transitions.TongUI generalizes multimodal web tutorials into GUI trajectories, MobileA3gent automatically annotates decentralized trajectories for federated training, and OS-Genesis synthesizes tasks from observed transitions.
  • Sparse-Reward Rollout Learning: Sparse-reward rollout learning closes the interaction-to-training loop by rolling out the current policy, automatically estimating rewards, and optimizing the GUI policy online.ZeroGUI uses VLM-based trajectory-level success, while MobileGUI-RL combines self-exploration, task filtering, trajectory-aware advantages, and composite rewards.
  • Limitations of Prior Rollout Learning: Existing rollout methods mainly use trajectory-level success or composite scalar rewards, making local credit assignment difficult because successful and failed rollouts can contain misleading actions.A single score can conflate task completion, efficiency, and action quality, while independent attempts do not provide reliable local retention signals.
  • MobileForge: MobileForge responds with MobileGym and HiFPO, using reachable target-app interaction, step-level process feedback, corrective hints, serialized attempts, and hint-contextualized step-level optimization.The system filters mastered tasks, preserves informative trajectories and local decisions, and reuses hints across attempts.

B Method Notation and Algorithm · C Pipeline Details · D Formal HiFPO Details

MobileForge organizes annotation-free adaptation as a target-app exploration-to-training loop and formalizes HiFPO as hint-contextualized, step-level policy optimization. The pipeline filters multi-attempt task groups, extracts informative local decisions, and trains on evaluated responses with group-relative feedback.

  • B Method Notation and Algorithm: MobileForge’s notation and Algorithm 1 summarize an annotation-free adaptation loop that maps target-app interaction into an adapted policy.The algorithm takes target apps E, policy πθ, and K attempts per task, and outputs adapted policy πθ′.
  • C Pipeline Details: The pipeline explores reachable GUI transitions and generates trajectory-grounded tasks from those transitions.These stages connect real target-app interaction to automatically constructed adaptation tasks.
  • C Pipeline Details: Each attempt is evaluated for outcome label z, step labels ℓ, and hint h, then the hint context is updated for later attempts.This feedback is carried forward across attempts rather than treating rollouts as isolated samples.
  • C Pipeline Details: Filtering removes mastered tasks, selects informative attempts, and extracts useful local steps before hint-contextualized step-level GRPO training.The main setting applies task-level success-rate filtering before best-trajectory and reasonable-step selection.
  • D Formal HiFPO Details: HiFPO aggregates corrective hints, samples subsequent attempts, and selects one informative attempt for each retained task.The selection prefers a successful attempt by Qk when one exists, and otherwise uses the highest-Qk attempt.
  • D Formal HiFPO Details: Tasks with SR(x) = 1 are removed, while tasks with SR(x) = 0 or 0 < SR(x) < 1 are retained for step extraction.The task-level success-rate filter is applied before best-trajectory selection so SR(x) is computed from the original multi-attempt group.
  • D Formal HiFPO Details: After filtering, each training example is a selected local decision rendered as a hint-contextualized prompt for response-group sampling.The old policy samples G responses for each prompt, and unparseable responses receive zero action reward.
  • D Formal HiFPO Details: HiFPO assigns adaptive GUI action rewards using type correctness and action-specific matching, then normalizes rewards within each response group to form group-relative advantages for clipped policy updates.The policy ratio and its clipped counterpart complete the GRPO-style optimization objective.

E Experimental Protocol Details

The experimental protocol evaluates MobileForge in-domain on AndroidWorld and out-of-domain on MobileWorld GUI-only, using two 8B base agents and automatically generated AndroidWorld adaptation tasks. Ablations examine the system’s feedback, optimization, filtering, evaluator, and curriculum components.

  • Benchmarks: AndroidWorld provides 116 in-domain tasks with Pass@1, Pass@2, and Pass@3 after MobileForge exploration, task mining, and HiFPO rollout collection.MobileWorld GUI-only provides a separate 117-task out-of-domain split, with no MobileWorld rollout, task, or feedback used for adaptation.
  • Base agents and adaptation scale: MobileForge adapts Qwen3-VL-8B-Instruct and GUI-Owl-1.5-8B-Instruct, generating 3,249 AndroidWorld-side candidate tasks from 527 source trajectory identifiers across 20 apps.Training scales are studied with 200-, 400-, and 900-task subsets.
  • Ablation scope: Ablations isolate corrective hints, hint-contextualized GRPO versus SFT, task-level success-rate filtering, final-decision evaluator choice, and trajectory-grounded curriculum coverage.These experiments target both rollout feedback and the resulting adaptation procedure.

F Detailed Ablation Results

This section presents the principal ablation trends and preserves complete numerical results for trajectory-level rollout experiments, including success, efficiency, critic, and curriculum statistics.

  • Figure 7 summarizes the principal ablation trends reported in the main paper.
  • The detailed tables retain raw success counts and percentages, rollout-efficiency statistics, critic precision and recall, and curriculum category counts.
  • Table 6 reports trajectory-level rollout ablations on 200 generated tasks with Qwen3-VL-8B, using corrective hints from previous attempts of the same task.

G Annotation-Free Adaptation Data Details

The AndroidWorld-side adaptation pool contains 3,249 candidate tasks grounded in 527 source trajectory identifiers from 20 apps.

  • Generated Adaptation Task Pool: 3,249 candidate tasks comprise the full AndroidWorld-side adaptation pool.The pool is generated by source app.
  • Generated Adaptation Task Pool: 527 source trajectory identifiers ground the generated adaptation tasks.
  • Generated Adaptation Task Pool: 20 apps serve as the sources for the AndroidWorld-side adaptation task pool.

H Exploration Phase Details · I Prompt Templates · J Adaptive GUI Action Reward

MobileGym builds annotation-free adaptation data through function-aware exploration, curriculum generation, hierarchical trajectory critique, and corrective hints. Its adaptive GUI action reward scores sampled actions against feedback-derived targets, gating argument credit on action-type agreement.

  • H Exploration Phase Details: MobileGym replaces random walks with function-aware exploration grounded in app metadata, screen parsing, visual grounding, and structured action generation.Activity-list anchors describe reachable app functions and screens without serving as demonstrations.
  • H Exploration Phase Details: At each state, an MLLM generates bounded, executable goals from the current screen and activity anchors, covering viewing, editing, searching, sharing, configuration, and lookup.The goal-generation design aims to make trajectories touch real app functionality.
  • H Exploration Phase Details: Depth-first exploration restores parent states by relaunching and replaying action prefixes, recording goals, screenshots, actions, execution metadata, and summaries into evidence pool Z.The collected pool is rich but unstructured.
  • I.1 Curriculum Generation Prompt: The MobileGym-Curriculum prompt jointly assesses exploration trajectories and generates 3–8 diverse tasks spanning app functionality, progressive difficulty, and 1–40-step lengths.Tasks avoid redundancy, emphasize core-functionality variations, and limit each functionality to at most three parameter variations.
  • I.2 MobileGym-Critic Prompts: MobileGym-Critic uses three hierarchical prompts to describe steps, judge completion and step reasonableness, and convert failures or inefficiencies into corrective hints.The final assessment synthesizes step descriptions, raw logs, and composite last-screen evidence while identifying feasibility and failure steps.
  • I.3 Hint-Guided Rollout Prompts: During HiFPO rollouts, corrective-hint context is appended to the task instruction before each attempt for both Qwen3-VL and GUI-Owl.The hint block summarizes prior mistakes, avoidance rules, suggested approaches, and important insights while preserving each agent’s native step format.
  • J Adaptive GUI Action Reward: The step-level GRPO reward parses each sampled response into a canonical action, compares it with the feedback-derived selected action, and combines configurable type and argument scores.Argument scoring is gated by type scoring, so an incorrect action type receives no parameter credit; malformed-call format score is logged but excluded from R.

K Training Details

The 900-task 8B HiFPO runs use a controlled, annotation-free AndroidWorld-side adaptation protocol shared across both base agents. Training rewards improve mainly through action-argument quality, while evaluation isolates responses to identical HiFPO signals and holds MobileWorld out.

  • Training dynamics: GUI-Owl-1.5-8B starts from lower overall reward but improves steadily, whereas Qwen3-VL-8B starts high and gains smaller but positive rewards.The curves indicate useful HiFPO learning signals for both a generalist VLM and a GUI-specialized base.
  • Training dynamics: Action-argument reward provides most sustained improvement for both agents, while action-type reward remains comparatively high.The overall reward tracks parameter quality because action-argument has weight 0.8, despite step-to-step variation from heterogeneous states and action schemas.
  • Training dynamics: The overall reward combines action-type and action-argument rewards with weights 0.2 and 0.8, respectively.Figure 10 reports overall, action-type, and action-argument curves for Qwen3-VL-8B and GUI-Owl-1.5-8B.
  • Reproducibility protocol: Both 8B agents train for four epochs on 900 generated AndroidWorld-side tasks, with no MobileWorld data entering adaptation.Prompts and responses are capped at 2,048 tokens; overlong prompts are filtered, and mastered tasks are excluded while corrective hints and best trajectories are retained.
  • Controlled comparison: The controlled comparison fixes task budget, filtering, rollout group size, optimizer, reward weights, and validation cadence while changing only the initial policy.Validation runs every 50 steps with greedy decoding and one response per prompt; MobileWorld is held out from adaptation and checkpoint selection.

K.1 Track-Completion Cases

Track-completion cases show that MobileForge-adapted agents recover from base-model failures by switching strategies, verifying conditions, preserving context, and completing downstream actions correctly across AndroidWorld and MobileWorld tasks.

  • AndroidWorld: On AndroidWorld’s Broccoli recipe-deletion task, ForgeQwen3-8B switches from repeated scrolling to targeted search and completes the remaining deletion.The base model makes partial progress before falling into repeated scrolling.
  • MobileWorld: On MobileWorld’s academic-calendar and email task, ForgeQwen3-8B verifies the Spring 2026 deadline, creates the calendar event, and proceeds to the email workflow.The base model selects an underspecified semester result and skips the calendar subtask.
  • MobileWorld: On MobileWorld’s invoice-and-email task, ForgeOwl-8B preserves invoice conditions, shares the document into Mail, and sends the correct recalculated total.The base model leaves the invoice context too early and sends an incorrect amount.
Loading 2606.19930v2…