Source-linked AI summary
Identifying User Goals from UI Trajectories
Omri Berkovitch, Sapir Caduri, Noam Kahlon, Anatoly Efros, Avi Caciularu, Ido Dagan
TL;DR
Inferring detailed user goals from UI activity could support personalization, but prior goal-recognition work mostly used language inputs and predefined categories. This paper defines goal identification from UI trajectories, proposes context-sensitive evaluation, and compares human and model performance on Android and web datasets, finding a significant performance gap.
Problem
Prior goal-recognition work mostly used natural-language inputs and predefined categories, while identifying goals from UI interactions could support assistance aligned with users’ preferences and objectives.
Source: §1 · p. 1Method
The paper defines natural-language goal generation from UI trajectories, adapts UI automation datasets by reversing input and output roles, and evaluates task-description matches in UI context.
Source: §1 · pp. 1–2, §3 · p. 3, §5.1 · p. 4Results
Takeaways & Limitations
The work introduces goal identification from UI trajectories and a proposed evaluation methodology for assessing it.
Source: §1 · p. 2, §7 · p. 5The study is limited to English-language interactions and Android and web environments, whose task distributions may differ from iOS and Windows.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Identifying underlying user goals and intents has been recognized as valuable in various personalization-oriented settings, such as personalized agents, improved search responses, advertising, user analytics, and more. In this paper, we propose a new task goal identification from observed UI trajectories aiming to infer the user's detailed intentions when performing a task within UI environments. To support this task, we also introduce a novel evaluation methodology designed to assess whether two intent descriptions can be considered paraphrases within a specific UI environment. Furthermore, we demonstrate how this task can leverage datasets designed for the inverse problem of UI automation, utilizing Android and web datasets for our experiments. To benchmark this task, we compare the performance of humans and state-of-the-art models, specifically GPT-4 and Gemini-1.5 Pro, using our proposed metric. The results reveal that both Gemini and GPT underperform relative to human performance, underscoring the challenge of the proposed task and the significant room for improvement. This work highlights the importance of goal identification within UI trajectories, providing a foundation for further exploration and advancement in this area.
1 Introduction
The paper frames goal identification from UI activity as a route to personalized assistance, addressing the gap between prior language-based classification and inferring natural-language goals from UI interactions. It defines the task, proposes context-sensitive evaluation, and compares human and model performance.
1 Introduction
- Understanding underlying user goals is framed as a cornerstone of user modeling for assistance aligned with preferences, habits, and long-term objectives.The flight-booking example illustrates suggesting a hotel and making travel dates visible in a calendar.Source: §1 · p. 1
- Prior goal-recognition work mostly used natural-language inputs and predefined categories, whereas this work identifies goals from UI interactions and describes them in natural language.The authors state that, to their knowledge, this is the first work to identify user goals from UI interactions with natural-language output.Source: §1 · p. 1
- The paper defines generating natural-language descriptions of user goals from multimodal UI trajectories, treating the task as the inverse of UI automation.Multiple goals can explain the same UI activity, making the task inherently ambiguous.Source: §1 · p. 1
- The paper introduces manual and automatic evaluation protocols for deciding whether predicted and gold task descriptions are paraphrases in a UI context.The criterion accounts for whether descriptions express the same intent within the environment.Source: §1 · p. 2
- Experiments on web and Android sessions compare human and state-of-the-art multimodal-model performance, finding substantial room for modeling improvement.The experiments adapt existing UI automation datasets for goal identification.Source: §1 · p. 2
2 Background and Related Work
Related work spans intent, activity, and plan recognition. The paper distinguishes its UI-trajectory goal-identification task from prior approaches that commonly classify text or sensor observations into predefined categories.
Intent Recognition
- Intent recognition traditionally classifies textual inputs, such as messages or utterances, into predefined intent categories.Prior examples also classify social-media posts for purchase intent or use external knowledge to interpret queries.Source: §2 · p. 2
- Multimodal intent-recognition work combines modalities such as images, text, audio, or video, but examples still classify inputs into intent topics or fixed categories.Examples include marketing intent in social news, intent in social posts, and 25 categories for TV-series data.Source: §2 · p. 2
Activity Recognition
- Activity recognition typically classifies physical actions from sensor observations, including skeletal data, accelerometers, and GPS traces.Examples include recognizing poses, nurse activities, and walking or sitting.Source: §2 · p. 2
Plan Recognition
- Plan recognition infers an overall plan or goal from relationships among observed actions, using either plan libraries or approaches based on in-domain knowledge.A hybrid example combines video-based activity detection with symbolic reasoning and a plan library.Source: §2 · p. 2
3 Task Definition
The task is to recover a user’s intended task from a UI trajectory, using the inverse framing of UI automation. The paper assumes the trajectory successfully fulfills the intent while recognizing that multiple intents may fit it.
3 Task Definition
- The task reverses UI automation’s input and output definitions, enabling reuse of datasets originally created for UI automation.The goal is to recover the original intent from the user’s observed trajectory.Source: §3 · p. 3
- A UI trajectory is a sequence of interaction steps, each pairing a UI snapshot with the user’s action, from which the system generates a natural-language task description.The paper assumes the observed trajectory successfully fulfills the underlying user intent.Source: §3 · p. 3
- A trajectory can fulfill multiple intents at different specificity levels, so the model is expected to predict the most likely intent reflected in the dataset distribution.A sushi booking trajectory, for example, could correspond to a specific restaurant, a broader area, or a cuisine-level goal.Source: §3 · p. 3
4 Evaluation Methodology
The methodology evaluates task descriptions against their UI trajectories, defining environment-specific task matching and using both human judgments and an automatic evaluator. Agreement checks provide evidence about the evaluation procedures’ reliability.
4 Evaluation Methodology
- The methodology compares predicted and gold task descriptions against the input trajectory, following definitions with human and automatic evaluation protocols.The evaluation asks whether the prediction matches the gold task description.Source: §4 · p. 3
4.1 Definitions
- Two tasks match when each mutually satisfies the other; a one-way satisfaction relation is a partial match, interpreted as a more general task in that UI environment.Satisfaction means every reasonable trajectory fulfilling one task also fulfills the other; the relation can depend on how the UI environment interprets terms.Source: §4.1 · p. 3
4.2 Human Evaluation Protocol
- Human annotators check whether the trajectory fulfills the prediction and whether predicted and gold tasks satisfy each other, while excluding validated noisy instances.Pairwise agreement among three authors was high: web Kappa values were 0.79 and 0.77, and Android values were 0.91 and 0.86, for fulfillment and satisfaction judgments respectively.Source: §4.2 · p. 3
4.3 Automatic Evaluation Metric
- GPT-4o’s automatic satisfaction judgments reached Kappa 0.48 against human evaluation, indicating moderate agreement and motivating manual evaluation.The authors describe model-based evaluation as potentially useful in development cycles while highlighting the need for manual evaluation.Source: §4.3 · p. 3
5 Datasets and Baseline Models
The section adapts UI-automation datasets by reversing intent and trajectory roles, using web and Android data to evaluate multimodal models on goal identification.
5.1 Evaluated Datasets
- UI-automation data supports goal identification by swapping the original intent and UI-trajectory input/output roles.Mind2Web and Android in the Zoo are the selected web and Android datasets.Source: §5.1 · p. 4
- Many candidate datasets lack high-level intent descriptions, suitable UI traces, or sufficiently generic GUI settings for this task.Examples include procedural-only instructions, domain-limited shopping tasks, and datasets without intent annotations.Source: §5.1 · p. 4
5.2 Models
- Gemini 1.5 Pro and GPT-4 process multimodal trajectories, with web screenshots marked around interacted elements and Android actions overlaid on screenshots.The models were guided through step-by-step Chain-of-Thought analysis.Source: §5.2 · p. 4
6 Experiments
Human and model evaluations reveal difficulty matching inferred goals to gold descriptions, with models still below human performance. The experiments also report dataset-specific human disagreements and a few-shot setback.
6.1 Human Performance Evaluation
- The human evaluation used 50 task descriptions per dataset, while model evaluation manually assessed 100 predictions per dataset and automatically evaluated the full test sets.The automatic sets included 1,013 Mind2Web and 506 AitZ examples.Source: §6.1 · p. 4, §6.2 · p. 4
- Human descriptions matched gold tasks more often in Mind2Web than AitZ, where gold descriptions were often more specific despite trajectories already fulfilling the task.Ambiguous trajectories also contributed to disagreements between human annotators.Source: §6.1 · p. 4, §6.2 · p. 4
6.2 Model Performance Evaluation
- Adding a single few-shot example deteriorated model results, likely because of large context size and differences between exemplar and test trajectories.The setting combines multiple images, textual inputs, and the model's thought process.Source: §6.2 · p. 4
- Gemini outperforms GPT on Mind2Web match scores but remains below humans, while the models perform comparably on AitZ.Table 2 covers manual and automatic fulfillment and match scores for model predictions against gold descriptions.Source: §6.2 · p. 4, Table 2 · p. 5
7 Conclusion and Future Work
The paper introduces goal identification from UI trajectories and an evaluation methodology, then identifies future work on improving models and testing downstream utility.
7 Conclusion and Future Work
- The paper introduces UI-trajectory goal identification and an evaluation methodology, finding a significant gap between humans and state-of-the-art multimodal models.The experiments use Android and web datasets.Source: §7 · p. 5
- Future work includes fine-tuning, improving visual understanding, testing downstream personalization and suggestion utility, and expanding to iOS and Windows.Source: §7 · p. 5
Limitations
The study is limited by its datasets and by its focus on Android and web environments.
Limitations
- Both datasets primarily cover English-language websites accessed in the U.S., and Mind2Web tasks remain within a single website.The authors note that real-world trajectories may involve interleaved tasks, evolving goals, or less-proficient users.Source: Limitations · p. 5
- The study covers only Android and web, whose task distributions may differ from those of iOS and Windows.Source: Limitations · p. 5
Ethical Considerations
The appendices describe privacy concerns and the datasets, action formats, and models used in the experiments. Mind2Web and Android trajectories pair screenshots with recorded interactions, while task-description prediction uses Gemini 1.5 Pro and GPT-4-Turbo.
Ethical Considerations
- Researchers should protect sensitive UI-activity data; on-device processing and anonymization are suggested as ways to mitigate privacy risks.The paper also urges researchers and developers to address these concerns to foster trust and responsible innovation.Source: Ethical Considerations · p. 5
A.1 Web
- Mind2Web trajectories pair screenshots with recorded actions, each comprising a target element and an operation such as Click, Type, or Select Option.The dataset spans popular websites across five top-level domains, and its actions were automatically produced during task recordings.Source: §A.1 · p. 6
A.2 Android
- AitW episodes pair screenshots with tap, drag, or typing actions, and the experiments present sequences of screenshots with actions drawn on them.Episodes were recorded on mobile devices running four Android versions; actions may include screen coordinates.Source: §A.2 · p. 6
B.1 Models Configuration
- Task-description prediction uses Gemini 1.5 Pro and GPT-4-Turbo, while GPT-4o is used for automatic evaluation.The reported Gemini version was updated in Vertex AI as of May 2024; GPT-4-Turbo is identified as gpt-4-turbo-2024-04-09.Source: §B.1 · p. 7
B.2 Data
The data evaluation sampled and verified examples, removing cases with unmet references or difficult-to-interpret web screens. The web and Android inputs were adapted to their differing visual and action formats.
B.2 Data
- Manual evaluation sampled over 100 examples across Android and web, checked reference and trajectory quality, and excluded cases where trajectories did not fulfill gold tasks.The review also removed web cases with improperly rendered screenshots or empty bounding boxes; 17% of Android examples and about 5% of web examples were rejected.Source: §B.2 · p. 7
C Error Analysis
- A car-selection example illustrates how annotators inferred price as the deciding factor even though the user chose the truck primarily for its 12-inch feature.The truck was also the cheapest, creating a competing explanation for the choice.Source: Figure 4 · p. 8
B.3 Web
- Mind2Web images had a median height of 4200 pixels, so the authors truncated them while keeping the interaction element visible and drew its bounding box to guide attention and resolve ambiguity.Twenty percent of images exceeded 7000 pixels in height; initial tests found oversized images introduced noise and negatively affected model performance.Source: §B.3 · p. 8
B.4 Android
- Because AitW lacks textual descriptions of tapped elements, experiments overlaid tap, drag, and typing actions on screenshot sequences and omitted annotations that confused models.The authors also removed the “Task Complete” label because it often confused models.Source: §B.4 · p. 8
C.1 Web
Across web and Android tasks, model errors varied by domain and included missed intent, excessive specificity, and difficulty interpreting action sequences. Human descriptions also diverged from gold references, while the appendix provides instructions for human annotation and model evaluation.
C.1 Web
- In analysis of 30 mismatched task descriptions, GPT started 12% of 1,000 predicted tasks with Navigate; over 20% of manually inspected errors involved misinterpreting intent, often producing broad descriptions lacking crucial details.Source: §C.1 · p. 8
- 72% of Mind2Web trajectories had all three task descriptions match, 24% had two match, and the rest had no match, suggesting most were probably non-ambiguous.Human disagreements with gold descriptions often reflected more generalized tasks when the trajectory did not reveal a less likely constraint.Source: §C.1 · p. 8
C.2 Android
- Both models handled Android’s General domain better than Web Shopping and Install; explicit-intent search queries helped, while action order and final-state interpretation caused difficulty.Google Apps tasks overlap with PixelHelp settings tasks, and some trajectories show a setting page without an action that reveals the intended change.Source: §C.2 · p. 9
- In Android shopping, Gemini could over-specify products while GPT missed the task’s purpose; ambiguous “open (install if not installed)” tasks also confused both models.Pre-installed apps could make an “open” prediction only partially satisfy the task in one direction.Source: §C.2 · p. 9
- Humans also struggled with ambiguous Install tasks and with inferring why a user chose a shopping result, but they read final setting states well when goals were specific.Visible search queries made General tasks minimally challenging for human annotators.Source: §C.2 · p. 9
D Instructions and Prompts
- The appendix presents instructions for human goal annotation, model trajectory-to-task prediction, and satisfaction evaluation; the Figure 6 instructions were the automatic evaluator’s core prompt.The prompts address whether a trajectory fulfills a task and whether two task descriptions satisfy each other.Source: Figure 6 · p. 10, Figure 7 · p. 11, Figure 8 · p. 12