A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility

1 Introduction

• MoTIF pairs natural-language commands with app screens, view hierarchies, and action coordinates, and includes both feasible and infeasible tasks.It adds fine-grained labels for why tasks are infeasible and follow-up questions for ambiguous queries.Source: §1 · pp. 2–3, Figure 1 · p. 2
• MoTIF evaluations find feasibility prediction and task automation challenging, with visual inputs and app exploration important to performance.Performance trends across seen and unseen apps suggest more in-app exploration during training is needed; qualitative failures also point to visual understanding.Source: §1 · p. 3
• The dataset supports feasibility classification and interactive task automation across mobile apps, including clicking, swiping, and typing.The contributions also include benchmarks for feasibility classification and task automation.Source: §1 · p. 3

2 Related Work

• Unlike static-image feasibility datasets such as VizWiz and bot-generated plausibility questions, MoTIF uses human-generated app requests whose feasibility may require exploration.MoTIF’s infeasible tasks remain relevant to the Android app category, unlike distinct visual failures such as blurred or random scenes in VizWiz.Source: §2 · p. 3
• MoTIF resembles interactive VLN datasets such as ALFRED [38], but uses a real, non-simulated domain rather than simulated environments [38].Its instructions target user goals through clicking, typing, or swiping.Source: §2 · p. 4
• Unlike MiniWoB and PhraseNode, MoTIF uses high-level goals and action sequences spanning multiple screens, with a median of eight visited screens.MiniWoB uses handcrafted HTML, while PhraseNode captures single-action commands on website home screens.Source: §2 · pp. 4–5

3 MoTIF Dataset

• Each demonstration records the app screen, backend view hierarchy, action type, and action location at every step.The hierarchy organizes screen elements as a tree and includes metadata such as class, resource ID, text, and clickability.Source: §3 · p. 5

3.1 Data Collection

• The collection framework selected 125 apps across 15 categories, gathered commands, then asked annotators to attempt tasks and flag infeasibility or ask clarifying questions.Task writers were asked for functional or navigational goals without prescribed task counts or structure.Source: §3.1 · pp. 5–6, Figure 3 · p. 6
• Category-clustered tasks were paired with other apps in the same Android category; isolated tasks remained app-specific.Tasks were clustered using mean FastText command embeddings and K-Means; this produced app-specific pairings for 40 apps and category-grouped tasks for 17 apps.Source: §3.1 · p. 6
• Annotators attempted tasks on physical Android phones and reported success; infeasible tasks were classified as impossible, unclear, or premature, with follow-up questions prompted when tasks could not be completed.The categories distinguish unavailable actions, under-specified requests, and tasks that may be possible but require figuring out how or completing other tasks first.Source: §3.1 · pp. 6–7, §3.2 · pp. 7–8, Table 2 · p. 7

3.2 Dataset Analysis

• MoTIF contains over 6.1k natural-language tasks across 125 apps, averaging 56 tasks per app and 5.6 words per task.Task lengths range from one to 44 words.Source: §3.2 · p. 7
• 29.2% of tasks were judged infeasible by at least five crowd workers, and 16.8% of infeasible tasks came from app-specific pairs.Some app-specific tasks were infeasible because app features changed or were dynamic.Source: §3.2 · p. 7

4 Task Feasibility Experiments

• The evaluation treats action demonstrations as oracle exploration and tests only app-task pairs with unanimous feasibility labels.The MLP predicts binary feasibility from commands and ablated app-environment features; its average F1 is computed over ten random initializations, with infeasible tasks as the positive class.Source: §4 · p. 8, §4.1 · pp. 8–9

4.2 Results

• CLIP embeddings of the task, view hierarchy, and screen image achieve the best feasibility-classifier result, an F1 score of 61.1.Combining view-hierarchy and screen features does not substantially outperform screen-image CLIP alone, and performs worse with LSTM aggregation.Source: §4.2 · p. 9
• The best classifier misclassifies 44% of infeasible tasks as feasible, while 5% of tasks are missed by being labeled infeasible.The authors note that attempting unsatisfiable requests can lead to unknown behavior, particularly for users unable to verify completion.Source: §4.2 · pp. 9–10
• CLIP representations of view-hierarchy elements outperform FastText, while Screen2Vec performs worse than CLIP and on par with FastText.The authors suggest Screen2Vec may not capture enough low-level hierarchy information.Source: §4.2 · p. 10
• Icon features are significantly less effective than ResNet and CLIP, and average icon features bring F1 close to zero.The authors suggest icon features may be too low-level or need improved aggregation.Source: §4.2 · p. 10
• Concatenating demonstration time steps tends to perform best across aggregation methods, but averaging beats LSTM when the best hierarchy and screen features are combined.The authors interpret the general concatenation trend as suggesting a need for sequential action-sequence representations.Source: §4.2 · p. 10

5 Task Automation Experiments

• The task is to interact with an app through actions such as clicking, typing, or swiping, and produce a sequence that completes the command.Evaluation includes app-seen and app-unseen test splits to study generalization to new environments.Source: §5 · p. 10, §5.1 · p. 11

5.1 Models

• Seq2Seq predicts actions and screen locations, while MOCA separates action-policy and interaction-grounding streams and adapts grounding to app-element classes.Seq2Seq replaces its original mask predictor with screen-point prediction; MOCA predicts among twelve app-element types and uses bounding-box localization.Source: §5.1 · pp. 10–11
• Seq2Act uses Transformers for action-phrase extraction and grounding, selecting app elements by similarity between phrase and element embeddings.It extracts action type, location, and input spans, then predicts an action and grounds it to a view-hierarchy element.Source: §5.1 · p. 11

5.2 Results

• Seq2Act achieves the highest performance across all metrics, while Seq2Seq and MOCA perform worse on MoTIF.Seq2Act's 70.6% complete Action + Ground accuracy on PIXELHELP is much higher than its MoTIF accuracy; the authors note PIXELHELP's click-only tasks and four test environments differ from MoTIF.Source: Table 4 · p. 12, §5.2 · p. 13
• Seq2Act can match task words to misleading app text, such as grounding “my profile” to an element labeled “My favorite.”The authors connect this text-matching bias to the need for visual input to correct direct matching between task text and view-hierarchy text.Source: §5.2 · p. 13, Figure 4 · p. 13
• Performance is worse on unseen app environments, and the authors suspect models learn too little about app elements outside demonstrated action sequences.None of the benchmark models include exploration; the authors suggest pretrained generic app features or exploration during training as possible future directions.Source: §5.2 · p. 13

6 Discussion

• The best feasibility predictor reaches only 61.1 F1, with demonstrations serving as oracle exploration; the authors identify learned exploration as a necessary next step.They also report that visual inputs help and suggest stronger app features and hierarchical visual-textual features as possible improvements.Source: §6 · p. 14
• Task-completion models have more difficulty grounding and perform more poorly in new app environments; better app-element representations or exploration beyond ground-truth actions may help.The authors note that MoTIF is smaller than pretraining datasets because collecting commands and feasibility labels is expensive and time-consuming.Source: §6 · p. 14
• Although collection avoids personally identifying information, downstream automation raises privacy concerns, so deployment may be limited to apps without login information and payment-related tasks can be denied.MoTIF contains no tasks requiring payment.Source: §6 · p. 14

7 Conclusion

• MoTIF introduces a mobile-app VLN dataset with natural-language tasks that may not be feasible, and the authors describe it as the first dataset to capture task uncertainty in interactive visual environments.The paper characterizes its linguistic and visual diversity as greater than prior work.Source: §7 · pp. 14–15
• The dataset supports research on feasibility prediction and task automation, and its results indicate new vision-language challenges for evaluating methods on realistic data.The conclusion states that MoTIF can be used to evaluate and improve current methods.Source: §7 · p. 15

8 MoTIF Collection

• MoTIF data collection used 34 annotators, with task-writing and task-demonstration work assigned to separate groups.Demonstrators were required to have Android experience; anonymized login information was provided to avoid collecting personally identifying information.Source: §8 · p. 18

8.1 Data Collection Interface

• The interface presented the task and anonymized credentials beside a remotely connected physical Android phone, while recording actions and app modalities.The interface captured the phone screen and recorded actions taken on the device.Source: §8.1 · pp. 18–19, Figure 5 · p. 19

8.2 Application List

• The selected apps span 15 Google Play categories; dating apps were included for task collection, but screen and view-hierarchy data for them will not be released for privacy.The authors state that no demonstrations of commands in dating apps were intended for collection.Source: §8.2 · pp. 19–20

8.3 Dataset Examples

• The supplementary examples show both feasible and infeasible app-task pairs alongside demonstrations, feasibility annotations, and follow-up questions.The demonstrations include rendered screens, app view hierarchies, and action localization.Source: §8.3 · p. 20, Figure 6 · p. 24, Figure 7 · p. 25

9.1 Natural Language Command Statistics

• Task vocabulary frequency has a long-tail distribution, and nearly all commands contain fewer than ten words, consistent with their focus on high-level goals.The vocabulary-frequency distribution follows Zipf’s Law.Source: §9.1 · p. 26

9.2 Word Cloud Visualizations

• Across categories, frequent command words tend to describe actions, while category-specific terms reflect app functions; task vocabularies are smaller for Food & Drink, Productivity, and Music & Audio than for Maps & Navigation, Weather, and Travel.The word-cloud visualizations encode more frequent words with larger lettering.Source: §9.2 · pp. 26–27, Figure 9 · p. 27, Figure 10 · p. 28

9.3 Annotator Feasibility Agreement

• Most app-task pairs with demonstrations have 90–100% feasibility-label agreement, including 296 pairs in the maximal-agreement bin.Agreement is defined as the majority-label vote count divided by all votes for an app-task pair.Source: §9.3 · p. 27, Figure 11 · p. 29

9.4 App Category Clustering Visualizations

• K-Means clusters of FastText task embeddings guide app-task pairing: an isolated Weather-app cluster remains app-specific, while other apps use category-clustered pairs.Commands near each cluster centroid are selected as representatives for demonstrations; the clustering uses K=5.Source: §9.4 · p. 29, Figure 12 · p. 30, Figure 13 · p. 31

• Ablations compare text (ET), resource-identifier (ID), and class (CLS) attributes individually and in combinations, with ET+ID performing best on average.The authors retain ET+ID for their main results.Source: §10 · p. 32

11.1 Test-time Evaluation of Seq2Seq and MOCA

• The test-time environment is an offline state-action graph built from merged demonstrations, with screens as nodes and action class and location defining transitions.Source: §11.1 · p. 33

11.2 Training Data Ablations

• Joint training on MoTIF and RicoSCA consistently outperforms MoTIF-only training across metrics, and app-seen accuracy is higher regardless of training data.Source: §11.2 · pp. 33–34

11.3 Test-time Language Input Ablations

• Seq2Seq performs best on action prediction with both language inputs, MOCA is similar with both or low-level input, and Seq2Act performs best with low-level instructions.MOCA shows less grounding degradation across ablations, potentially because app-type prediction narrows its grounding space; Seq2Act does not support both inputs.Source: §11.3 · pp. 34–35

11.4 Generalization of Natural Language Commands across Apps

• Models generalize tasks across applications better than they generalize to new app environments, while performance is higher when the app is seen and the task is unseen.The app-seen, task-unseen split allows models to learn the app environment and task instruction independently during training.Source: §11.4 · pp. 35–36