PaperlayerView exact revision on arXiv

Source-linked AI summary

From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks

Yuchen Sun, Chenglin Cai, Gongjie Zhang, Tianyu Xia, Quyu Kong, Panrong Tong, Zhengwen Zeng, Long Chen, Steven Hoi, Chongyang Zhang, Yue Wang

arXiv:2609.30887v1cs.AI

TL;DR

Mobile GUI agents typically navigate screen by screen, even when a task-relevant screen may be directly reachable through an app deeplink. GUI-Hopper combines deeplinks for navigation with GUI control, supported by a verified catalog and training for link selection and fallback. The paper reports improved task success and interaction efficiency on MobileWorld and performance on MobileWorld-Real, with gains persisting without deeplinks.

  • Problem

    GUI-only navigation requires agents to traverse intermediate screens, motivating whether they must use GUI actions for every navigation step.

    Source: §1 · p. 1

  • Method

    GUI-Hopper uses a real-device-verified catalog of described deeplinks and trains an agent to select links for navigation while retaining GUI actions and fallback.

    Source: §1 · p. 2

  • Results

    GUI-Hopper completes more MobileWorld tasks with fewer interaction steps than a matched GUI-only agent across three model scales, retains gains without deeplinks, and performs well on MobileWorld-Real.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support combining deeplinks for higher-level navigation with GUI actions for fine-grained operations and fallback.

    Source: §1 · pp. 1, 3

    Deeplink coverage is limited to screens apps expose for external access, and the evaluation is restricted to Android.

    Source: §5 · p. 10

Abstract

from arXiv · show

Mobile GUI agents complete tasks using GUI actions like taps and swipes. These actions are broadly applicable across applications, but reaching a navigation interface. A single deeplink call can replace a sequence of screen-by-screen GUI actions. We therefore introduce hybrid interaction, using deeplinks for direct navigation and GUI actions for other on-screen operations and fallback. To enable this, we discover candidate deeplinks through static analysis, validate them on real devices, and describe their observed landing screens. This process creates a verified and grounded deeplink catalog that pairs each working deeplink with a description of its landing screen. Using this catalog, we introduce GUI-Hopper, a improves task success in commercial applications on real devices, further demonstrating the benefits of hybrid interaction.

1 INTRODUCTION

Screen-by-screen GUI navigation can make long mobile tasks take many steps, while deeplinks can open task-relevant screens directly. The paper combines deeplinks with GUI control, builds a verified catalog, and trains GUI-Hopper to choose between the two; evaluations report better task completion and interaction efficiency.

1 INTRODUCTION

  • Screen-by-screen navigation requires interaction with intermediate screens, lengthening trajectories and creating additional opportunities for error.This motivates asking whether agents must use GUI actions for every navigation step.Source: §1 · p. 1
  • A single deeplink can open a task-relevant screen directly, while GUI actions retain fine-grained control and provide fallback for the rest of the task.The Figure 1 example contrasts a GUI-only route requiring 15+ steps with hybrid navigation.Source: §1 · p. 1, Figure 1 · p. 2
  • The catalog pipeline executes candidate deeplinks on real devices, filters failures or meaningless destinations, and describes each retained link's observed landing screen.It can search for parameter values when declarations do not specify them.Source: §1 · p. 2
  • GUI-Hopper learns when to invoke a deeplink and when to continue through GUI control, using negative augmentation and online reinforcement learning.At inference, it retrieves links by landing-screen description and falls back to GUI interaction if no suitable link is available or verification fails.Source: §1 · pp. 2–3
  • GUI-Hopper completes more MobileWorld tasks with fewer interaction steps than a matched GUI-only agent, with the advantage across three trained model scales and on real devices.Its gains also persist when evaluated without deeplinks.Source: §1 · p. 2

2 RELATED WORK

Prior mobile GUI agents use screen-by-screen GUI navigation, while tool-augmented approaches add reusable skills or application functions. The paper positions app-native deeplinks as another route and studies their systematic mining, validation, and use with GUI actions.

2 RELATED WORK

  • Existing mobile GUI agents include prompted multimodal-model frameworks and end-to-end trained GUI models, but both still navigate screen by screen.Recent benchmarks emphasize long tasks, cross-application workflows, and real-device interaction, making trajectory efficiency relevant alongside task success.Source: §2 · p. 3
  • Agent-built tools can be brittle, while platform interfaces can be dependable but have coverage limited by developer adoption; app-native deeplinks offer a different path.The paper distinguishes its approach by automatically mining and validating deeplinks, training agents to choose between them and GUI actions, and verifying use at inference.Source: §2 · p. 3
  • Prior deeplink research covers generation, testing, and security, but does not study how a trained mobile agent combines deeplinks with GUI actions.The cited literature also reports that production OEM assistants already invoke deeplinks to bypass screen-by-screen navigation.Source: §2 · p. 3
  • Figure 2 organizes the approach into device-verified catalog mining, hybrid-action training, and inference-time retrieval with landing checks and GUI fallback.The figure describes the pipeline components rather than a comparison with prior work.Source: Figure 2 · p. 4

3 METHOD

GUI-Hopper combines deeplink actions with GUI control in a partially observable task formulation. Its method builds a verified, described catalog, trains action selection and recovery, and uses a harness to retrieve, verify, and parameterize deeplink calls.

3.1 PRELIMINARIES

  • The hybrid action space combines atomic GUI operations with catalog deeplink calls parameterized for the current task, and observations include screenshots and deeplink feedback.The policy selects actions from the task instruction, current observation, and recent interaction history.Source: §3.1 · p. 4

3.2 DEEPLINK MINING AND DESCRIPTION

  • Catalog mining filters non-exported or unresolved candidates, tests resolvable links on real devices, checks parameter effects, and removes duplicates and transient targets.The resulting catalog contains 1,251 verified entries across the 26 commercial apps selected for MobileWorld-Real.Source: §3.2 · p. 5
  • Each catalog entry describes its observed landing screen and parameters, and stores an invocation template and expected Activity for call construction and landing verification.Descriptions also clarify what remains to be done after reaching a screen, since landing there does not necessarily complete the task.Source: §3.2 · p. 5

3.3 TRAINING IN A HYBRID ACTION SPACE

  • Hybrid trajectory synthesis teaches deeplink chaining and GUI continuation, while negative examples train fallback when links are irrelevant and recovery after an unsuitable call.Deeplink dropout withholds useful links; wrong-call injection deliberately leads to an irrelevant screen before the recovery sequence.Source: §3.3 · p. 5
  • Online reinforcement learning combines call-level landing verification with trajectory-level success and length signals to refine deeplink selection and efficient completion.The reward gives failed trajectories zero; among successful ones it increases with call score and shorter length below the reference.Source: §3.3 · p. 6

3.4 AGENT HARNESS

  • The harness constructs a task-specific set of at most 10 deeplinks by matching navigation goals to applications and ranking entries semantically and lexically.It reuses the compact candidate set throughout the task rather than exposing the full catalog at every step.Source: §3.4 · p. 6
  • The harness checks both call execution and whether the observed landing Activity matches the catalog's expected Activity, making failure explicit for GUI recovery.A successful system-command return alone does not establish that the intended screen was reached.Source: §3.4 · p. 6
  • During ordinary app use, the harness captures reusable parameter values from application Intents and associates them with catalog entries for later calls.Examples include dates, locations, and item identifiers observed in interaction history.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments compare GUI-Hopper with GUI-only agents in simulation and on physical devices, and test the contributions of training stages and deeplink supply. GUI-Hopper improves task success and interaction efficiency, including when deeplink candidates vary or are unavailable.

4.1 SETUP

  • MobileWorld evaluates all 117 tasks across 13 built-in apps, while MobileWorld-Real evaluates 119 tasks across 26 selected apps.Each simulated task is run three times per setting; up to 10 catalog candidates are retrieved at inference.Source: §4.1 · pp. 7–8
  • The three-scale comparison pairs GUI-only and GUI-Hopper training on the same tasks and configurations, differing in deeplink actions and associated feedback.The evaluated scales are 4B, 9B, and 35B-A3B.Source: §4.1 · p. 7

4.2.1 MOBILEWORLD

  • Across three MobileWorld model scales, GUI-Hopper achieves higher success rates than the GUI-only agent and reduces average steps by 15.26–21.30%.It also has shorter successful trajectories at every scale.Source: §4.2.1 · p. 8
  • On MobileWorld, GUI-Hopper’s hybrid-SFT checkpoints retain higher success rates and fewer GUI steps than the GUI-only agent when evaluated without deeplinks.The authors suggest hybrid-action training may encourage intermediate-goal and screen planning that also helps GUI-only use.Source: §4.2.1 · p. 8

4.2.2 MOBILEWORLD-REAL

  • On MobileWorld-Real, GUI-Hopper outperforms the GUI-only agent in task completion and trajectory efficiency on commercial applications running on physical devices.Table 2 presents the real-device benchmark comparison.Source: Table 2 · p. 8, §4.2.2 · p. 9

4.3 ABLATIONS

  • Online reinforcement learning raises success from 57.83% to 60.11% and reduces average steps across all trajectories from 22.52 to 20.91.Deeplink supply is unchanged in this training-stage comparison.Source: §4.3 · p. 9

4.3.2 EFFECT OF DEEPLINK SUPPLY

  • For Qwen3.7-plus, task-matched deeplinks reduce trajectory length by 16.25%, but broader Ten and Full catalogs lower success to 52.99%.The authors attribute the degradation to decision noise that the base model cannot filter.Source: §4.3.2 · p. 9
  • GUI-Hopper maintains nearly identical success rates (∼63%) and step counts (∼21.7) across GT, Ten, and Full deeplink supplies.Ten trails the oracle GT setting by 0.29 percentage points; the authors describe the policy as robust to noisy candidate pools.Source: §4.3.2 · p. 9, Table 4 · p. 9

5 CONCLUSION

The paper presents GUI-Hopper, which combines deeplinks for navigation with GUI control and uses a verified catalog grounded in observed landing screens. Evaluations report improved success and shorter trajectories across tested scales, with gains persisting when deeplinks are withheld; the approach is limited to externally exposed screens and Android.

5 CONCLUSION

  • GUI-Hopper combines app-native deeplinks for high-level navigation with fine-grained GUI control, using a mined, real-device-verified catalog of landing-screen descriptions.The policy is trained through hybrid trajectory synthesis and online reinforcement learning to invoke deeplinks or fall back to GUI actions.Source: §5 · p. 9
  • Across MobileWorld and MobileWorld-Real, evaluations report higher task success and shorter trajectories across tested model scales, with gains persisting when deeplinks are withheld at inference.The conclusion connects withheld-deeplink gains to enhanced underlying task-planning capabilities.Source: §5 · pp. 9–10
  • Without system privileges, deeplink coverage is limited to screens applications expose for external access, and the evaluation is restricted to Android.Future work proposed includes system-level integration and dynamically enriching catalogs with user-specific parameters captured during live interaction.Source: §5 · p. 10

A MINING PIPELINE: IMPLEMENTATION, PROMPTS, AND CATALOG STATISTICS

The appendix details how candidates are extracted, checked, and consolidated into a verified catalog, then explains the catalog format and how agents use its deeplinks alongside GUI actions.

Mining pipeline

  • Mining extracts declared candidates, removes non-exported targets, checks resolvability without launching apps, and tests remaining entries on real devices.Parameterized entries are retained only when changing a value visibly changes the landing screen.Source: A MINING PIPELINE: IMPLEMENTATION,… · p. 14
  • Deduplication consolidates verified entries using device calls, screenshot and text comparisons, and command-level checks, while uncertain cases remain separate.Transient content instances are excluded; one map search screen was reachable through 11 entries spanning five domain variants and three URI schemes.Source: A MINING PIPELINE: IMPLEMENTATION,… · p. 14

Catalog statistics

Mining pipeline

  • The verification prompt guides a vision-language model to propose parameter values, invoke candidates, assess landing screens, and write catalog descriptions; a second prompt uses screenshots to help deduplicate entries.Prompts also exclude tracking- or session-bound parameters, and the calling program selects retained entries under fixed rules.Source: A MINING PIPELINE: IMPLEMENTATION,… · pp. 14–15
  • The deduplication prompts treat entries as the same feature when they land on the same page with interchangeable parameter values, but separate different tabs or business objects.When uncertain, the prompt keeps entries separate rather than risking a wrong merge.Source: A MINING PIPELINE: IMPLEMENTATION,… · p. 15

B DEEPLINK CATALOG FORMAT AND AGENT PROMPT

  • A catalog entry gives the agent a landing-screen description and parameter guidance, while harness-side fields support execution and landing verification.The AMap example opens route results for the specified origin and destination; starting navigation may still require a GUI action.Source: B DEEPLINK CATALOG FORMAT AND AGENT… · pp. 15–16, Figure 3 · p. 16
  • The evaluation prompt presents selected deeplinks as function-calling schemas and tells the agent to verify task completion through the GUI.The illustrated prompt retains 10 candidate deeplinks.Source: B DEEPLINK CATALOG FORMAT AND AGENT… · pp. 16, 18
  • Deeplinks open or prefill screens rather than completing submissions, so the agent must use GUI actions to finish and verify actions such as saving or sending.Tool descriptions specify whether a final submit remains necessary.Source: B DEEPLINK CATALOG FORMAT AND AGENT… · p. 18

C TRAINING CONFIGURATION

The training configuration aligns GUI-only and hybrid supervised fine-tuning settings, then specifies online reinforcement-learning sampling, limits, and deeplink call credit.

SFT configuration

  • GUI-only and hybrid models use the same supervised fine-tuning configuration at each scale, differing only in their training trajectories.The configuration uses a 0.05 warmup ratio, bf16 precision, and no sequence packing.Source: C TRAINING CONFIGURATION · p. 18

Online reinforcement learning

  • Each online reinforcement-learning iteration starts from the SFT checkpoint and samples 117 tasks with eight trajectories per task, limiting each trajectory to 50 interaction steps and context to 20,480 tokens.Policy updates use a global batch size of 512 and asymmetric clipping with ϵlow = 0.2 and ϵhigh = 0.28.Source: C TRAINING CONFIGURATION · p. 18
  • A task-relevant deeplink earns 0.3 partial credit when invoked but none of its invocations reaches the expected Activity.The policy initializes from the supervised fine-tuning checkpoint.Source: C TRAINING CONFIGURATION · pp. 18–19

D AGENT HARNESS IMPLEMENTATION

The harness executes catalog deeplinks, checks the observed landing Activity, retrieves a task-specific candidate set, and captures reusable parameter values from app use.

Execution and feedback

  • The harness fills a catalog command template with supplied parameters, dispatches an Android Intent, and checks whether the expected Activity reaches the foreground.It waits up to 10 seconds when the target app is foregrounded, but stops within 4 seconds if another package is foregrounded; mismatches or failures are returned to the agent for GUI fallback.Source: D AGENT HARNESS IMPLEMENTATION · p. 19

Task-level retrieval

  • Task-level retrieval combines embedding and BM25 rankings and returns up to 10 application-matched deeplinks; manually annotated task requirements yield 98.3% COMP@10 coverage.Lexical scores affect ranking but do not filter candidates.Source: D AGENT HARNESS IMPLEMENTATION · p. 19

Capture in use

  • The harness captures parameter values from observed Intents for later reuse, excluding values used for tracking or tied to a particular session.Observed values can include order identifiers, saved addresses, or travel dates.Source: D AGENT HARNESS IMPLEMENTATION · p. 19

E FURTHER ANALYSIS

Under GUI-only evaluation, GUI-Hopper retains a success-rate advantage across model scales and has comparable trajectory efficiency at 9B. A trajectory example illustrates a shorter route to needed information.

E.1 GUI ABILITY WITHOUT DEEPLINKS

  • GUI-Hopper achieves higher task success rates than the GUI-only trained model at all three scales under GUI-only evaluation, with comparable 9B trajectory efficiency and a slightly lower mean length.Figure 4 compares successful-trajectory length distributions.Source: Figure 4 · p. 19, §E.1 · p. 19
  • In the 35B-A3B example, GUI-Hopper finds a recipient’s email address in two steps versus 21 for the baseline, while both use nine steps to compose and send the email.GUI-Hopper returns to the resume earlier; the savings come from choosing a more direct path to the needed information, not from fewer email-form actions.Source: §E.1 · p. 20

E.2 EFFICIENCY ACROSS TASK STRUCTURES

GUI-Hopper improves success on both single-app and multi-app tasks while reducing trajectory length, with larger length reductions on multi-app tasks. Its cross-application use combines deeplinks with GUI actions across workflow stages.

E.2 EFFICIENCY ACROSS TASK STRUCTURES

  • Multi-app tasks create repeated opportunities to bypass GUI navigation because agents must reach relevant screens in several apps.The evaluation includes 55 single-app tasks and 62 multi-app tasks, with three runs per task.Source: §E.2 · p. 20
  • Trajectory length falls by 18.4% on multi-app tasks and 11.2% on single-app tasks, while GUI-Hopper improves success in both groups.Table 6 compares the Qwen3.5-9B GUI-only model and GUI-Hopper; success gains are larger for single-app tasks, while trajectory reductions are larger for multi-app tasks.Source: §E.2 · p. 20, Table 6 · p. 20
  • In a cross-application interview-invitation example, GUI-Hopper opens Downloads and prefills an SMS by deeplink, then uses GUI actions to read the resume and send it.The example illustrates deeplinks and GUI actions working together across multiple stages of a workflow.Source: §E.2 · p. 20

E.3 TASK EXECUTION TIME

GUI-Hopper reduces mean execution time on MobileWorld-Real across three model scales. MobileWorld records also show lower durations, but differing timing methods prevent a controlled speedup estimate.

E.3 TASK EXECUTION TIME

  • Mean execution time on MobileWorld-Real falls by 2.9–11.0% across the three model scales.Each comparison uses the same tasks at a given scale and counts at most the first 50 trajectory steps; MobileWorld-Real timing covers successful and unsuccessful trajectories.Source: §E.3 · p. 20, Table 8 · p. 21
  • MobileWorld records show lower durations for GUI-Hopper, but the models' different timing methods preclude a controlled speedup estimate.Tables 7 and 8 report mean durations over successful and unsuccessful trajectories.Source: §E.3 · p. 20, Table 7 · p. 20

F CASE STUDIES

The case studies show trajectory compression through deeplink-assisted navigation and better GUI planning, alongside a recovery case where reaching the expected screen did not ensure correct task parameters.

F CASE STUDIES

  • GUI-Hopper completes the interview-invitation task in five steps versus 26 for the GUI-only baseline, using deeplinks for Downloads and SMS prefilling and GUI actions to read the resume and send the message.The sequence shows deeplinks shortening navigation and text entry while GUI actions handle the remaining operations.Source: F CASE STUDIES · p. 21, Figure 5 · p. 21
  • Without deeplinks, both 35B-A3B models complete the interview-email task; GUI-Hopper takes 14 steps versus 32 for the GUI-only baseline, returning earlier to the resume to obtain the recipient’s email address.In the illustrated stage breakdown, finding the address takes 21 steps for the baseline and two for GUI-Hopper.Source: F CASE STUDIES · p. 21
  • On the calendar task, GUI-Hopper reaches the expected Activity with an incorrect date, then recreates the event through GUI actions in a 19-step trajectory versus 18 for the baseline.The GUI recreation takes 11 steps; the trajectory does not show removal of the earlier wrong-date event.Source: Figure 7 · p. 22, F CASE STUDIES · p. 23

Related papers

Loading 2609.30887v1…