META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI

1 Introduction

• Traditional TOD systems depend on publicly accessible or TOD-specific APIs and predefined API-related slots, limiting applicability and generality.Source: §1 · p. 1
• GUI offers a more general interaction interface because users can complete smartphone tasks by interacting directly with the interface rather than a parametric API.Prior GUI-operation studies mainly address single queries or step-by-step operations, whereas real scenarios involve multi-turn interaction without clear execution instructions.Source: §1 · pp. 1–2
• GUI-TOD performs multi-turn tasks directly through GUI operations on real APPs, avoiding TOD-specific APIs and domain-specific schemas.The system is designed for application across different apps and domains rather than optimization for a specific app.Source: §1 · p. 2
• 82.74% action completion rate shows GUI-TOD outperforming heuristic baselines by a large margin.Source: §1 · p. 2
• The paper contributes a GUI-TOD system, the META-GUI benchmark, and experiments validating the importance of multimodal and history information.The reported experiments indicate that GUI-TOD is promising but still requires further development.Source: §1 · p. 2
• META-GUI provides dialogues and GUI operation traces from real Android APPs as a benchmark for GUI-TOD.The traces include screenshots, Android view hierarchies, and actions.Source: §1 · p. 2

• The system comprises an Action Executor that predicts GUI actions and a Response Generator that produces replies from execution results.Source: §2 · pp. 2–3
• GUI-TOD directly operates real app GUIs through multiple actions instead of relying on task-oriented dialogue APIs.Source: §2 · p. 3
• Unlike conventional dialogue managers, GUI-TOD handles task-specific slot values during execution rather than using a predefined domain ontology.Source: §2 · p. 3
• Action prediction uses the GUI trace and dialogue history to select the next action, while response generation uses the trace and dialogue context to produce a reply.Source: §2 · p. 3
• GUI-TOD represents tasks with six physical actions—click, swipe, input, enter, clear, and back—and one virtual end action.Source: §2 · p. 3

3 Meta-GUI Creation

• META-GUI contains 1,125 dialogues across weather, calendar, search, taxi, hotel, and restaurant domains, with GUI operation traces for multi-turn tasks.The dataset includes multiple apps for calendar and weather to increase diversity.Source: §3 · p. 3

3.1 Collecting GUI traces

• The collection used two stages: annotators executed existing calendar, weather, and search dialogues, then created dialogues and traces for hotel, restaurant, and taxi tasks.The first stage used dialogues extracted from SMCalFlow, while the second used randomly generated app conditions and turn constraints.Source: §3.1 · pp. 3–4
• Each GUI trace records operations together with post-operation screenshots and Android view hierarchies.The view hierarchy organizes GUI content and includes item types and bounding boxes.Source: §3.1 · p. 4

3.2 Data Review

• Manual review checked dialogue-trace consistency, invalid operations, and redundant operations; after correction or discard, the dialogue-level pass rate was about 63.6%.The final reviewed collection contained 1,125 dialogues.Source: §3.2 · p. 4

3.3 Post-processing

• When valid Android hierarchies were unavailable, OCR generated pseudo-layout files, from which clickable items were extracted using text attributes, item types, and bounding boxes.Click targets were located by comparing recorded click positions with item bounding boxes.Source: §3.3 · p. 5

3.4 Data Analysis

• The dataset contains 4,684 turns, averages 5.30 images per turn and 23.80 items per image, and has average utterances of 8 words.TextView and ImageView are the most frequent item types.Source: §3.4 · p. 5
• Click is the most frequent action and clear the least frequent; TextView and Button are the item types most often targeted by clicks.The average response and input-action lengths are 9 and 3 words, respectively.Source: §3.4 · p. 5

4 Model Design

• The proposed architecture contains an encoder, image feature extractor, multimodal fusion module, and either an Action Module or Response Module.Source: §4 · p. 5, Figure 5 · p. 6

4.1 Action Model

• The encoder represents dialogue history and texts from items on the last screen, while regional image features are extracted using Faster R-CNN and ROI pooling.Source: §4.1 · p. 5
• The fusion module concatenates item text with corresponding image features, assigns zero features to dialogue tokens, and applies a Transformer encoder.Source: §4.1 · p. 5
• The Action Module predicts seven action types and their parameters, using classifiers for action type, input spans, target items, and swipe direction.Source: §4.1 · pp. 5–6
• Action histories are added as special tokens, while screenshot histories are recurrently encoded with attention and the final history feature replaces the current image features.Source: §4.1 · p. 6

4.2 Response Model

• The Response Module uses a Transformer Decoder to generate responses from execution results and dialogue context without action histories.Source: §4.2 · p. 7

5.1 Data Preprocess

• 18,337 action-level data points are split into training, development, and test sets at an 8:1:1 ratio.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 Experiment Setup

• The evaluation uses action and turn completion rates, accuracy, exact match, F1, and BLEU to assess action, parameter, and response prediction.Source: §5.2 · p. 7

5.3 Experiment Result

• BERT outperforms LayoutLM and LayoutLMv2, while adding multimodal information improves overall performance from 52.08% to 53.96%, mainly through action type, target item, and swipe direction prediction.Source: §5.3 · pp. 7–8
• Adding action histories raises completion rate from 52.08% to 55.42%, while adding screenshot histories raises it from 53.96% to 55.62%.Source: §5.3 · p. 8, Table 3 · p. 8
• 56.88% completion rate is achieved by m-BASH, which combines multimodal information, action histories, and screenshot histories with BERT.Source: §5.3 · p. 8, Table 3 · p. 8
• The response model also benefits from multimodal information and screenshot histories, while BERT outperforms LayoutLM and LayoutLMv2 on response generation.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Generality

• App- and domain-generality experiments show reasonable performance, with app generality comparable to the main LayoutLM results and applicability to new apps or domains without modification.Source: §5.4 · pp. 8–9, Table 5 · p. 9

6.1 Natural Language Commands on GUI

• Natural-language GUI-command research maps utterances to website operations, while Google Duplex supports only limited websites and is described as more a unified interface than a general GUI dialogue system.Source: §6.1 · p. 9

6.2 Programming by Demonstration on GUI

• Programming-by-demonstration systems generate scripts from GUI operations, but handcrafted rules and XML analysis make them sensitive to GUI changes and exceptions.Source: §6.2 · p. 9

6.3 Visual Dialogue

• Visual dialogue models reason over image contents and dialogue history, whereas the paper’s task additionally requires understanding interactions between user interfaces.Source: §6.3 · p. 9

• GUI-TOD replaces traditional TOD-specific API calls with GUI operations on real APPs.Source: §7 · p. 9
• META-GUI contains dialogues and GUI traces for benchmarking GUI-based task-oriented dialogue systems.Source: §7 · p. 9
• The proposed model shows promising results on META-GUI.Source: §7 · p. 9

• META-GUI contains dialogues and GUI operation traces but excludes cases where agents cannot complete the requested task.Source: Limitations · pp. 9–10
• The models used are too large to run on mobile phones.Source: Limitations · p. 10
• Future work will add unsuccessful-task cases and compress the models for mobile deployment.Source: Limitations · p. 10

• The dataset uses four weather apps, three calendar apps, and one app for each of the remaining four domains.Source: A Details of Apps · p. 13
• Table 6 lists the applications used in the dataset and their turn counts.Source: Table 6 · p. 13, A Details of Apps · p. 13
• The total number of turns exceeds the actual turn count because one turn may involve several apps.Source: Table 6 · p. 13, A Details of Apps · p. 13

C Example of View Hierarchy

• The annotation system is illustrated in Figure 6, while Figure 7 demonstrates the correspondence between a screen element and its view-hierarchy entry.Source: Figure 6 · p. 13, Figure 7 · p. 13

B Annotation System

• The annotation system provides Dialog, Screen, Action, and Screenshot boxes for viewing dialogues, inspecting screens, controlling the phone, and recording operations.Source: B Annotation System · p. 13

C Example of View Hierarchy

• The view hierarchy represents a screen’s GUI elements in a structured hierarchy and identifies the corresponding highlighted element.Source: Figure 7 · p. 13

D Data Review

• Manual review checks trace-dialogue alignment, invalid operations from errors or misoperation, and redundant GUI actions.Examples include verifying scheduled times, detecting mistaken clicks or swipes, and removing repeated screenshots.Source: D Data Review · p. 13

E Example of the generated target during collecting

• The collecting process includes examples of generated targets.Source: Figure 8 · p. 14

F Examples of Item types

• The dataset contains 18 item types, with representative items highlighted by red borders in Table 7.Source: F Examples of Item types · p. 14, Table 7 · p. 14

G Examples of parameter predictions

• Parameter prediction examples cover target-item selection, input-text spans obtained from tokenized dialogue, and direction prediction.The target-item example contrasts the current screenshot with the screenshot after clicking the item.Source: G Examples of parameter predictions · p. 15, Figure 9 · p. 15

H Case study

• In click-action cases, m-BASH identifies the true target items while BERT errors because text alone cannot distinguish selected conditions; images and history compensate for this limitation.Source: H Case study · p. 15, Figure 10 · p. 15