PaperlayerView exact revision on arXiv

Source-linked AI summary

Mapping Natural Language Instructions to Mobile UI Action Sequences

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge

arXiv:2005.03776v2cs.CLcs.LG

TL;DR

The paper studies how to ground natural-language instructions in executable mobile UI actions. It decomposes the task into phrase extraction and UI-object grounding, creates three datasets to support evaluation and scalable training, and reaches 70.59% complete-match accuracy on PIXELHELP.

  • Problem

    The paper addresses the gap between retrieving multi-step natural-language instructions and automatically executing their corresponding action sequences on mobile touch-screen UIs.

  • Method

    The approach uses Transformer-based action phrase extraction and contextual UI-object grounding, supported by PIXELHELP, ANDROIDHOWTO, and RICOSCA datasets.

  • Results

    70.59% complete and 89.21% partial accuracy are obtained for matching ground-truth action sequences on PIXELHELP.

  • Takeaways & Limitations

    The decomposition allows progress in action span extraction or UI grounding to improve full-task performance.

  • Takeaways & Limitations

    Grounding from learned hidden-state representations remains challenging when learning from unpaired instruction-action data because synthetic characteristics do not manifest in PIXELHELP.

Abstract

from arXiv · show

We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it. For full task evaluation, we create PIXELHELP, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator. To scale training, we decouple the language and action data by (a) annotating action phrase spans in HowTo instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces. We use a Transformer to extract action phrase tuples from long-range natural language instructions. A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions. Given a starting screen and instruction, our model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PIXELHELP.

1 Introduction

The paper frames mobile UI task automation as grounding natural-language instructions in executable touch-screen action sequences. It decomposes this challenge into extracting action phrases and grounding them to UI objects, supported by three new datasets and strong PIXELHELP performance.

  • Motivation: Mobile UI automation requires mapping multi-step natural-language instructions to executable action sequences with little user intervention.The setting is especially relevant to visually or situationally impaired users who may not be able to access UI details easily.
  • Approach: The proposed approach separates action phrase extraction from grounding descriptions to UI objects on each screen.Transformers extract operation, object, and argument descriptions, while a contextual screen encoder matches descriptions to UI objects.
  • Datasets: Three datasets support full-task evaluation, phrase extraction, and grounding-model training.PIXELHELP contains naturally occurring instructions and executed action-screen sequences; the other datasets decouple language and action data for scalable training.
  • Results: 85.56% accuracy is achieved on completely matching ground-truth span sequences in ANDROIDHOWTO.This result uses a Transformer with spans represented by sum pooling.
  • Results: 89.21% partial and 70.59% complete accuracy are obtained for matching ground-truth action sequences on PIXELHELP.The complete-match result evaluates whether the entire predicted action sequence matches the ground truth.

2 Problem Formulation

The task models an instruction as descriptions of one or more UI actions and seeks to generate executable actions across successive screens. Each action combines an operation, target object, and optional argument, while the model factorizes phrase extraction from screen grounding.

  • Task definition: Given a multi-step instruction and starting screen, the goal is to generate executable actions over a sequence of UI screens.Executing prior actions transitions the interface from the initial screen to the screen associated with each subsequent step.
  • Action representation: Each action consists of an operation, a UI object, and an additional argument when required.Examples include Tap or Text operations applied to a button or icon, with entered text as an argument for Text.
  • Screen representation: Each screen contains UI objects and their structural relationships, often represented by an Android View hierarchy.The target object is selected from the variable-sized set of objects on the current screen.
  • Language representation: Action descriptions are represented as tuples whose operation, object, and argument components correspond to token spans in the instruction.The instruction may describe multiple actions, producing a sequence of description tuples.
  • Model factorization: The model factorizes prediction into identifying action-description tuples and grounding each description to an executable action given the current screen.This decomposition separates language-dependent phrase prediction from screen-dependent action grounding.

3 Data

The paper builds PIXELHELP for full-task evaluation and separate datasets for phrase extraction and grounding. Human-executed mobile tasks provide natural action sequences, while web annotations and synthetic UI commands enable scalable training.

  • 3.1 PIXELHELP Dataset: PIXELHELP excludes instructions requiring additional user input or physical-button actions that cannot be executed on the emulator.Examples include choosing an app to uninstall and holding the physical Power button.
  • 3.1 PIXELHELP Dataset: PIXELHELP pairs English Pixel Phone instructions with logged human actions and screen sequences from a mobile emulator.The logger records touch-event types, manipulated objects, view hierarchies, and the target action on each screen.
  • 3.1 PIXELHELP Dataset: PIXELHELP contains 187 multi-step instructions across general, Gmail, Chrome, and Photos tasks, with two to eight steps per instruction.The dataset is reserved for evaluating full task performance.
  • 3.2 ANDROIDHOWTO Dataset: ANDROIDHOWTO supplies annotated action-description spans from web instructions because no existing dataset supported mobile-UI phrase extraction.Each action component is marked by annotators as a span of words, with three-annotator labeling and reported agreement at instruction and tuple levels.
  • 3.2 ANDROIDHOWTO Dataset: ANDROIDHOWTO contains 32,436 data points from 9,893 How-To instructions, covering sequences of one to 19 actions.Its train, validation, and test splits contain approximately 8K, 1K, and 900 instructions, respectively.
  • 3.3 RICOSCA Dataset: RICOSCA synthetically pairs grounded UI actions with varied natural-language commands using structured and visual UI knowledge.Commands vary operation wording and describe targets by name/type, absolute location, or relative location.
  • 3.3 RICOSCA Dataset: RICOSCA is derived from Android UI screens containing object names, types, and bounding-box positions.The source Rico corpus was filtered to 25K unique screens after checking view hierarchies against screenshots.

4 Model Architectures

The model extracts operation, object, and optional-argument phrase spans from multi-step instructions, then grounds them to executable actions using screen-specific UI representations.

  • 4.1 Phrase Tuple Extraction Model: Phrase Tuple Extraction encodes instruction tokens and decodes tuples whose spans identify each action’s operation, object, and optional argument.Missing phrases use a special span encoding, and previously decoded tuples condition each decoding step.
  • 4.1 Phrase Tuple Extraction Model: The extractor represents variable-length spans and scores their alignment with task-specific query vectors for operation, object, and argument phrases.Separate trainable parameters produce query vectors for the three phrase types, while span representations support overlapping action descriptions.
  • 4.1 Phrase Tuple Extraction Model: The phrase extraction model is trained with softmax cross-entropy loss between predicted and ground-truth spans across phrase-tuple sequences.At each decoding step, the model consumes previously decoded tuples, using either concatenated or summed element representations.
  • 4.2 Grounding Model: The grounding model maps each extracted phrase tuple to an operation type, a screen-specific UI object, and an argument when present.A grounded action tuple can then be automatically executed; the model uses the screen at each instruction step.
  • 4.2 Grounding Model: Grounding assumes the operation is described without screen information and that mobile-task arguments occur only for Text operations.Operation probabilities are modeled from the operation description, while object selection uses an alignment-based softmax over screen objects.
  • 4.2 Grounding Model: UI-object representations combine object content with spatial and structural position, and grounding is trained using cross-entropy losses for object and operation prediction.Content includes object name and type, while spatial position uses the object’s top, left, right, and bottom screen coordinates.

5 Experiments

Experiments evaluate phrase-tuple extraction and full instruction grounding using separate datasets, span representations, and UI-object encoders. The Transformer screen encoder reaches 70.59% complete-match and 89.21% partial-match accuracy on PIXELHELP, while analyses identify remaining data- and representation-related limitations.

  • Datasets and Metrics: The experiments train and validate on ANDROIDHOWTO and RICOSCA, then evaluate full grounded action sequences on PIXELHELP using Complete and Partial Match.Complete Match requires identical tuple sequences; Partial Match measures the fraction of ground-truth steps matched.
  • Tuple Extraction: Area attention provides a small boost over start-end span representations on the ANDROIDHOWTO test set, although complete-match performance still has considerable headroom.The compared representations include area attention, start-end concatenation, and its generalized form.
  • Grounding: 70.59% Complete Match and 89.21% Partial Match accuracy are achieved by the Transformer screen encoder on PIXELHELP.These results are statistically significant across five runs and establish a strong baseline for the new dataset while leaving headroom.
  • Grounding: GCN-based grounding methods perform poorly relative to the Transformer, while both still outperform the heuristic baseline of 62.44% partial and 42.25% complete match.The results indicate that contextual encodings of information from other UI objects are important, whereas view-hierarchy distance can introduce noisy bias.
  • Analysis: Phrase extraction failed on 14 PIXELHELP tasks, producing extra steps in 11 and incorrect steps in 3, with no skipped steps.The authors relate these errors to differing language styles across the three datasets.

6 Related Work

The work extends language grounding from desktop and web interfaces to executable actions on mobile user interfaces, drawing connections to semantic parsing, human-robot interaction, and language-conditioned navigation.

  • Prior work grounded natural language in desktop or web interfaces, while this work targets executable actions on mobile user interfaces.
  • Related semantic-parsing research includes generating executable outputs such as SQL queries.
  • The task also relates to human-robot interaction, where dialogue results in robot actions.
  • Operating user interfaces resembles language-conditioned navigation, where agents execute instructions as sequences of movements.

7 Conclusion

The paper presents an initial approach to grounding natural language instructions in mobile UI actions and identifies directions for improving both extraction and grounding.

  • The work is an important first step toward grounding natural language instructions to mobile UI actions.
  • Because the task is decomposed, progress in either action span extraction or grounding can improve full-task performance.
  • Future directions include span-identification and multitask-learning innovations, reinforcement learning, and direct grounding from hidden state representations.
  • The work provides a technical foundation for investigating language-based human-computer interaction experiences.

A Data

The appendix describes how ANDROIDHOWTO action phrases were annotated through a web interface, including annotator procedures, tuple fields, and instruction statistics.

  • Annotation procedure: ANDROIDHOWTO action phrase spans were labeled by contractors using a web interface that displayed the instruction and collected action phrase tuples.
  • Annotation procedure: Annotators selected an operation type from Click, Swipe, Input, and Others, then marked action-verb and object-description spans.
  • Dataset characteristics: Instructions contained 19–85 tokens, with a median of 59, and described 1–19 actions, with a median of 5.

B Computing Span Representations

The appendix compares span-representation algorithms and explains efficient computation of sum-based and weighted span vectors, including parallel tensor operations and cumulative sums.

  • Representation choices: Three span-representation types were evaluated: sum pooling, Start-End Concat, and its generalized form.
  • Sum pooling: Sum pooling represents each span by summing token encodings and computes all span representations in constant time using summed area tables.
  • Start-End Concat: Start-End Concat represents a span from the encodings of its start and end tokens, while the generalized form adds weighted token embeddings and a span-length feature.
  • Weighted span sums: Weighted span sums are computed in parallel from hidden states and token embeddings, with a maximum span length hyperparameter M.
  • Weighted span sums: The weighted computation forms token weights, applies element-wise multiplication to embeddings, and divides cumulative weighted sums by cumulative weights.
  • Efficient computation: Sum vectors are computed by forming an integral image through cumulative summation along the sequence dimension.

C Details for Distance GCN

Distance GCN defines soft adjacency by applying a Gaussian kernel to structural distances between UI objects in the view hierarchy, then uses standard GCN computation.

  • Distance-based adjacency: Structural distance between view-hierarchy objects determines their soft adjacency strength through a Gaussian kernel.The distance is computed between object pairs based on the view hierarchy tree.
  • Distance-based adjacency: The adjacency formulation uses d(o_i, o_j) for the distance between objects and σ as a constant.
  • GCN computation: After defining soft adjacency, Distance GCN follows the typical GCN computation.

D Hyperparameters & Training

The models use 128-dimensional embeddings and hidden states, task-specific Transformer or GCN configurations, and distinct training schedules for phrase extraction and grounding.

  • Shared settings: 128-dimensional token embeddings and hidden states were used for all models after tuning embedding depth, hidden size, learning rate, schedules, and dropout.Increasing dimensionality did not improve accuracy and slowed training.
  • Phrase tuple extraction: Phrase tuple extraction used six-layer, eight-head Transformers with 10% dropout and a learning rate rising to 0.001 before exponential decay.Models were trained for 1 million steps.
  • Grounding models: Grounding GCNs used six ReLU layers, 10% dropout, and a peak learning rate of 0.0003.Filter-1 GCN and Distance GCN were trained for 250K steps.
  • Grounding models: The Transformer screen encoder used six layers with 30% ReLU dropout, 40% attention dropout, and 20% preprocessing dropout.Its peak learning rate was 0.001, and it was trained for 250K steps on the same hardware.
Loading 2005.03776v2…