PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions

Juyong Lee, Woogyeol Jin, Kimin Lee

arXiv:2609.06792v1cs.AI

TL;DR

Hybrid GUI-and-tool action spaces for Android agents remain underexplored because creating tools requires substantial human labor. DroidTool generates and verifies Python tools over application states through an agentic workflow with relational tests, and its augmented agents achieved higher task success with fewer interactions than GUI-only agents across three benchmarks.

  • Problem

    Hybrid Android agents remain underexplored because crafting mobile tool actions requires substantial human labor.

  • Method

    DroidTool generates Python tools operating on application states and verifies them through agentic proposal, implementation, relational testing, and repair workflows.

  • Results

    4.47%p higher task success and approximately 20.05% fewer interactions were achieved on average than GUI-only agents across AndroidWorld, B-MoCA, and MobileSafetyBench.

  • Takeaways & Limitations

    Verified self-generated tools provide additional action options associated with improved proficiency and efficiency across the evaluated Android-agent benchmarks.

  • Takeaways & Limitations

    DroidTool still requires human-curated inputs such as Developer Documents, and its tools are generated before downstream task execution.

Abstract

from arXiv · show

Android agents using a hybrid action space that combines GUI actions and tool actions (e.g., accessing application data via APIs) remain largely underexplored, mainly due to the excessive effort required to create tools. To address this gap, we introduce DroidTool, a framework for augmenting the agents with self-generated tools, which are realized as Python functions operating on application states (e.g., a database). To create tools with minimal human labor, DroidTool employs an agentic workflow featuring stages: proposal, implementation, test generation and execution, and repair. Notably, when testing the created tools for verification, it constructs relational tests across relevant tools for natural preparation of appropriate test preconditions and improved test coverage, rather than testing each tool separately. The GUI agents augmented with the generated tools achieved approximately 4.47%p higher performance with approximately 20.05% fewer interactions than the GUI-only agents, averaged across representative benchmarks: AndroidWorld, B-MoCA, and MobileSafetyBench.

1 Introduction

DroidTool addresses the labor barrier limiting hybrid GUI-and-tool action spaces by generating tools through an agentic workflow and testing related tools together. Across three benchmarks, augmented agents improved task success while using fewer interactions than GUI-only agents.

  • Motivation: Tool actions can replace long GUI sequences by directly operating on application states such as databases.The hybrid approach complements GUI actions with an alternative interface for application operations.
  • DroidTool: DroidTool generates Python tools for application states through proposal, implementation, verification, and iterative repair stages.Verified tools become additional action options that agents can invoke alongside GUI actions.
  • Verification: Relational tests connect relevant tools so one tool can establish preconditions for another, improving verification coverage and tool reliability.For example, creation can prepare an entity for reading and subsequent deletion.
  • Results: 4.47%p higher task success and 20.05% fewer interactions were achieved on average across AndroidWorld, B-MoCA, and MobileSafetyBench versus GUI-only agents.The result summarizes performance and interaction efficiency across the three representative benchmarks.

2 DroidTool

DroidTool augments GUI agents with tools that programmatically operate on structured Android application states. An LLM-agent workflow proposes, specifies, implements, tests, repairs, and registers these tools, using relational test sequences to prepare realistic preconditions.

  • Hybrid action space: DroidTool forms a hybrid action space by combining GUI actions with programmatic tool actions.Tool outputs are returned in the agent’s historical context for subsequent decisions.
  • Tool scope: The tools operate on application databases, file storage, and SharedPreferences, and are implemented as Python functions executed through ADB.Supported operations include database read, create, update, and delete, plus setting get and set operations.
  • Generation workflow: The generation workflow uses proposal, structured specification, Python implementation, test generation and execution, repair, and registration stages.Rule-based checks validate generated code and test suites, while failed tools enter an iterative repair loop.
  • Relational verification: Relational test suites sequence related tool calls so earlier outputs establish preconditions for later tests.A create-read-delete sequence can generate an entity, verify it, and remove it before registration.
  • Registration: Tools that pass their associated tests are registered as additional action options and exposed through metadata for function calling.Registered metadata describes each tool’s name, inputs, outputs, and short description.

3 Experiment

DroidTool was evaluated against GUI-agent baselines across AndroidWorld, B-MoCA, and MobileSafetyBench, alongside an ablation of verification strategies. Self-generated tools improved average performance and reduced interaction demands, while relation-aware testing registered more tools and supported stronger downstream results.

  • Benchmarks: The evaluation covers 335 tasks across 29 applications from AndroidWorld, B-MoCA, and MobileSafetyBench.The benchmarks include general, application-setting, and safety-related scenarios.
  • Baselines: The baselines compare GUI-only, GUI+Skill, GUI+CLI, and DroidTool agents using GUI actions plus self-generated tool actions.All baselines use Gemini-3.5-flash, including DroidTool’s tool-generation process.
  • Self-generated tools: 4.47%p higher average task performance and approximately 20.05% fewer interactions were obtained by DroidTool versus GUI-only agents.Average interactions decreased from 8.13 to 6.50, with especially large reductions on complex tasks such as alarm setting in B-MoCA.
  • Baselines: GUI+Skill did not yield consistent improvements, whereas GUI+CLI showed modest improvements over GUI-only.This comparison contextualizes the stronger reported results for self-generated tools.
  • Verification strategies: Relation-aware testing registered 113 tools, compared with 101 for Unit and 82 for Separate, out of 121 proposed tools.The ablation compares mocked offline unit tests, separate emulator tests, and ordered relational emulator tests.
  • Verification strategies: Relation agents achieved a 79.20% average benchmark success rate, exceeding Unit at 78.61% and Separate at 76.22%.They also achieved a 90.20% tool-call success rate while making 517.0 tool calls.

4 Conclusion

DroidTool augments Android GUI agents with self-generated tool actions operating on application states. Its workflow proposes, implements, verifies, and repairs tools, which agents can invoke alongside GUI actions; evaluations showed improved performance and efficiency.

  • DroidTool augments Android GUI agents with tool actions operating on application states.
  • Its agentic workflow automatically proposes, implements, verifies, and repairs new tools before promoting them as additional action options.
  • Verified tools are used through function calling alongside GUI actions on three established benchmarks.
  • The augmented agents showed improved performance and efficiency across the evaluated benchmarks.

Limitations

The paper identifies limitations in adaptive tool management, remaining human input, generation cost, and GUI-based verification. It positions these as directions for future research rather than resolved capabilities.

  • DroidTool agents currently rely on tools generated before executing downstream tasks.Dynamic generation, selection, and removal of tools during execution remain future directions.
  • The workflow still requires human-curated inputs such as the Developer Document.Source-code inspection and runtime analysis are proposed to reduce this remaining dependence on human expertise.
  • Tool generation incurs costs, although reuse can amortize them after creation.Analyzing and improving these costs remains an open direction.
  • Automated GUI-based testing of generated tools remains underexplored.Benchmark evaluation partially verifies user-visible effects, but stronger GUI testing could improve tool stability.

Broader impacts

The work frames automatically generated tools as a way to improve mobile-agent reliability and efficiency while complementing, rather than replacing, user oversight and application security controls.

  • Generated tools may make mobile agents more robust to interface changes, reduce execution costs, and improve accessibility.
  • Deployment should include least-privilege access, explicit consent for consequential actions, secure data handling, and auditable execution logs.
  • Generated tools are intended to complement user oversight and application security controls.

A Related work

Related work spans mobile-device agents and benchmarks, GUI agents with hybrid action spaces, and automatic code generation with verification and supporting tool infrastructure.

  • Mobile device-use agents: Mobile-device-agent research progressed from imitation and reinforcement learning to LLM-powered agents and broad capability benchmarks.
  • GUI agents with hybrid action space: Recent GUI-agent studies add complementary modalities such as command-line interfaces and Python programs, while exploring GUI–tool action selection and interleaving.
  • Code generation and verification: Code-generation research has translated API specifications into executable tools and developed application-specific tool libraries and servers.

B Tool generation workflow

The workflow uses LLM-based agents across five stages to automatically generate new tools.

  • LLM-based agents perform design, specification, implementation, test generation, and repair stages.

B.1 Workflow procedure

The workflow uses Developer Documents to identify application state targets, design lifecycle-oriented tool families, and specify, implement, and verify state operations. Its documents encode state representations, access procedures, and consistency constraints while avoiding unsupported details.

  • Developer Documents provide compact, human-curated descriptions of application states, schemas, access procedures, and consistency requirements.They reduce the need to inspect full application source code and infer storage semantics independently.
  • A state surface is an underlying representation or programmatic interface through which application state is exposed.Candidate surfaces include databases, files, SharedPreferences, content providers, and system services.
  • The workflow specifies state representations, access procedures, and consistency or recovery constraints while requiring unsupported surfaces and lifecycle behavior to remain unintroduced.Document rules cover state-model structure, access ordering, deletion boundaries, and consistency categories.
  • Entities are classified as primary, secondary, or auxiliary to prioritize targets and handle related state relationships.Auxiliary entities represent derived state, metadata, or lightweight relation records supporting other entities or preserving consistency.
  • The design agent proposes coherent lifecycle tool families for target entities, such as create, read, update, and delete operations.Setting-related targets instead use complementary get and set operations.

B.2 Workflow configuration details

The experiments configure the workflow with deterministic LLM generation, bounded regeneration and repair attempts, and per-test-mode verification loops.

  • The workflow uses gemini-3.5-flash with temperature 0.0 for proposal, specification, implementation, test generation, and repair.
  • Tool implementation and test generation allow one initial generation plus at most three regeneration attempts after rule-based validation failure.Proposal and target specification use a single model call per application.
  • Verification-repair runs independently for each test mode for at most eight rounds, registering a target only when all declared tools pass.Actionable execution failures are sent to the repair model in each round.

C Benchmark evaluation setup

The benchmark evaluation setup covers Android GUI-agent scaffolding, benchmark metrics, and agents whose action spaces are augmented with tools or skills.

  • The evaluation describes Android GUI-agent scaffolding, including base LLM configuration and prompts eliciting GUI-action performance.
  • The setup specifies benchmark evaluation details, including the metrics used to assess agent performance.
  • The evaluation demonstrates agents with skills and agents whose action spaces are augmented with tool actions.

C.1 GUI agent scaffolding

The GUI agent receives screenshots, task instructions, and interaction history, then emits a structured description, progress summary, next action, and executable function call at each step.

  • Interaction loop: At each interaction step, the agent receives the task instruction, current device screenshot, and textual history of previous steps.History records screen descriptions, task progress, selected actions, and returned tool results when applicable.
  • Interaction loop: The response contains the visible-screen description, task-progress context, next action, and one executable function call.The environment executes the call, captures the resulting screen, and repeats until completion or termination.
  • Action interface: GUI actions include screen touch, swipe, text input, button presses, application opening, waiting, answering, and termination.The available action set is instantiated according to each benchmark, with additional benchmark-specific actions and rules where applicable.
  • Benchmark rules: Benchmark-specific rules govern termination, task procedures, and safety-sensitive behavior across AndroidWorld, B-MoCA, and MobileSafetyBench.For unsafe or consent-sensitive tasks, MobileSafetyBench directs the agent to refuse or ask consent rather than use ordinary failure termination.

C.3 Agent with skills

The skill-based agent uses app-specific guidance generated from exploratory GUI trajectories, while separating descriptive application knowledge from verified, reusable procedures.

  • Skill generation: App-specific skills are natural-language guidance generated by the agent and inserted into the system prompt for later task solving.The skill guide is constructed through exploration and skill-generation stages.
  • Exploration: Exploration covers startup state, navigation, layout, controls, settings, entity lifecycles, safe inspection routes, and state-changing controls.The predefined patterns are ux_overview, settings_configuration, and entity_lifecycle.
  • Evidence policy: The exploration agent must rely on visible GUI evidence and avoid converting attempted, blocked, contradictory, or incomplete transitions into verified procedures.Failed and maximum-step rollouts can inform descriptive knowledge but cannot alone establish a complete skill.
  • Skill generation: Skill generation transforms rollout metadata, statuses, screen descriptions, GUI actions, and execution results into structured application knowledge, skills, and unsupported workflows.The generated document contains general application knowledge and only complete, repeatable, visibly verified GUI skills.
  • Verified skills: A verified skill requires a grounded starting condition, repeatable GUI route, visible success condition, and evidence-backed recovery and safety information.The output includes preconditions, procedure, success checks, failure recovery, safety, and evidence fields.

C.5 Benchmark setup details

The evaluation compares GUI-only, skill, CLI, and hybrid action-space variants under matched benchmark conditions, using task success as the main outcome across three Android-agent benchmarks.

  • Evaluation design: All agent variants use the same task manifest, instructions, initialization procedure, model configuration, and interaction budget; they differ in available information or actions.The GUI-only baseline receives mobile_use, while other variants receive additional skills or action capabilities.
  • Evaluation design: Each evaluation episode restores or initializes benchmark state, applies task-specific setup, returns an initial screenshot, and updates the agent after each action.The interaction protocol executes actions within episodes beginning from benchmark-defined initial states.
  • Tasks and benchmarks: The study evaluates 116 AndroidWorld task types, 119 B-MoCA tasks, and 100 daily-scenario MobileSafetyBench tasks.B-MoCA excludes Instagram and Walmart-related tasks, while MobileSafetyBench excludes robustness-related tasks and other specified subsets.
  • Metrics: Task success rates are the principal metric, with MobileSafetyBench using safety score for high-risk tasks and proficiency score for other tasks.AndroidWorld and B-MoCA use environment-signaled task success with the stated termination handling.
  • Tool-generation scope: Tool generation targets persistent, locally accessible application states and covers named applications across AndroidWorld, B-MoCA, and MobileSafetyBench.Applications whose primary state is managed on remote servers are excluded from tool-generation candidates.

D Testing Method Variations

The study compares unit, separate, and relation-aware verification approaches for generated tools, emphasizing executable tests, deterministic fixtures, meaningful assertions, and valid preconditions.

  • Testing variations: The testing variations include relation-aware test-case generation, whose details are provided separately from unit and separate testing procedures.The paper identifies relation-aware generation as a distinct verification method.
  • Unit testing: Unit testing generates standard-library Python test modules that directly invoke each generated tool without requiring an emulator.Tests mock external boundaries while exercising the implementation’s local parsing, querying, conversion, and return-shaping logic where possible.
  • Unit testing: Generated unit tests cover normal behavior and at least one relevant error or edge case, with validators checking structure and executable test content.Tests must invoke the named public tool and include observable assertions rather than merely inspecting source or callability.
  • Separate testing: Separate testing executes tools independently on an emulator, restoring the same base snapshot and deterministic application initialization for every case.Cases cannot use state or outputs produced by other planned cases.
  • Separate testing: Separate-test generation requires at least one case per declared public tool and marks cases ready only when required selectors are guaranteed in the initialized baseline.Cases needing unavailable pre-existing objects or selectors are classified as fixture_unavailable.
  • Assertions and validation: Ready cases use machine-checkable assertions over declared return schemas, while intentional negative cases specify expected errors and omit regular assertions.Validators check response structure, tool coverage, identifiers, parameters, assertion compatibility, and fixture rules.

E Additional experimental results

Additional experiments examine generated tools, agent augmentation variants, and verification strategies. Results show useful specialization, but also schema, safety, command, and reliability limitations that relation-aware verification can mitigate.

  • Exemplary generated tools: Generated Clock tools provide self-contained create, list, update, and delete operations over the alarms database.Their implementation inspects the schema, preserves SQLite metadata and sidecars, and cleans dependent rows during deletion.
  • Tool verification and repair: DroidTool repaired Broccoli tools whose initial implementations did not conform to the concrete database schema.A missing required favorite column caused create_recipe to fail with a NOT NULL constraint error before repair; subsequent recipe calls succeeded.
  • Agents with skills: Skill augmentation raised high-risk-task proficiency from 20.00% to 22.00% but reduced safety from 68.67% to 64.00% on MobileSafetyBench.The result indicates that improved task execution can coincide with less appropriate handling of unsafe requests.
  • Agents with CLI action: CLI augmentation was mixed: it enabled direct application-state manipulation in some tasks, but failures from syntax, paths, intents, and permissions often increased interactions or hindered completion.Across three evaluation runs, 573 of 684 CLI commands succeeded, corresponding to an 83% success rate.
  • Agents with unverified tools: Unverified tools sometimes improved benchmark success but had the lowest tool-call success rate, 73.41%, because of constraint violations, schema mismatches, and malformed commands.Verification detected these errors during generated-test execution, reducing their occurrence in downstream calls.
Loading 2609.06792v1…