Source-linked AI summary
Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, Brad A. Myers
TL;DR
End users often express conditional tasks with vague concepts, while existing systems offer limited support for teaching agents new concepts. PUMICE combines natural-language instruction, conversation, and demonstrations grounded in mobile-app GUIs; in a 10-participant lab study, users completed all four tasks and found the system natural and useful.
Problem
End users may use unclear concepts when specifying task conditionals, and supporting concept learning remains a challenge for domain-independent task automation.
Source: Introduction · pp. 1–2Method
PUMICE lets users describe tasks in natural language, resolve ambiguities through conversation, and teach concepts and procedures using third-party mobile-app GUIs and demonstrations.
Source: Introduction · p. 2, Example Scenario · p. 5Results
All participants completed all 4 tasks; participants also praised PUMICE’s naturalness and low learning barriers and found interactive concept resolution useful.
Source: Results · p. 10, Discussion · p. 10Takeaways & Limitations
PUMICE demonstrates multimodal interactions for helping users explain abstract task concepts through smaller concepts and demonstrations in third-party apps.
Source: Conclusion · p. 11The parser does not handle concept relations, references, synonyms, antonyms, or implicit conjunctions and disjunctions in utterances.
Source: Limitations and Future Work · p. 11
Abstract
from arXiv · showhide
Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multi-modal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability.
Introduction
End users struggle to specify conditionals because programming structures can differ from how they think about familiar tasks, while natural-language systems also struggle with ambiguity and vague concepts. PUMICE combines natural-language instruction and GUI demonstrations so users can describe tasks at a high level and iteratively clarify concepts and procedures.
Motivation
- End users find conditionals difficult partly because their mental models of tasks differ from the way programming languages represent conditional structures.Natural-language programming systems that require fixed, programming-like expressions can impose learning barriers and cognitive demands.Source: Introduction · p. 1
- Users often use ambiguous or vague concepts such as “cold weather” and “heavy traffic,” which an agent may not understand using its language techniques and predefined ontology.The paper identifies support for teaching concepts and explaining unclear concepts as a challenge for natural-language programming.Source: Introduction · p. 2
Approach
- The approach combines natural-language programming with programming by demonstration, representing actions as GUI operations and concepts through GUI contents.It supports tasks across domains when those tasks can be performed with existing third-party mobile apps.Source: Introduction · p. 2
- A persistent knowledge graph records user-taught actions, concepts, and facts, supporting fine-grained reuse and improving the parser’s understanding over time.Users explicitly introduce procedural and declarative knowledge through mixed-initiative conversations.Source: Introduction · p. 2
- PUMICE’s multimodal interface combines demonstrations, pointing, and spoken commands, with visual and verbal guidance through the programming process.The guidance addresses challenges including omitted else statements and difficulty identifying GUI objects or describing them.Source: Introduction · p. 2
- PUMICE lets users describe tasks and conditionals naturally, then recursively clarify ambiguities, explain concepts, and define procedures through conversation.Users can also teach concepts by referring to third-party app GUIs and teach procedures by demonstrating in those apps.Source: Introduction · p. 2
Background and Related Work
The related work spans natural-language programming, programming by demonstration, and multimodal systems combining language with demonstrations. PUMICE uses conversation as the primary modality and demonstrations to ground concepts, values, and procedures.
Natural Language Programming
- Natural-language systems that constrain users to programming-like expressions can have high learning barriers, while conversational clarification lets users build complex commands incrementally.PUMICE adopts multi-turn conversation to handle ambiguity and vagueness.Source: Natural Language Programming · p. 3
- Unlike conversational instructable agents limited to one or a few predefined domains, PUMICE supports learning from third-party mobile apps across task domains where the relevant apps are available.Users reference app GUI information to create concepts and demonstrate procedures; prior agents’ supported building blocks can be unclear.Source: Natural Language Programming · p. 3
Programming by Demonstration
- PUMICE uses programming by demonstration to let users define concepts through app GUI references and teach procedures through demonstrations.The passage describes PBD as a natural, low-learning-barrier approach to end-user development.Source: Programming by Demonstration · p. 3
- PUMICE is described as the only compared GUI-based PBD system that learns concepts as generalized knowledge and supports creating conditionals from natural-language instructions.Sikuli supports conditionals through a scripting language, which the passage says is unsuitable for end users without programming expertise.Source: Programming by Demonstration · p. 3
The Multi-Modal Approach
- Demonstrations communicate what a user does, while language can express why and how; grounding language in GUI demonstrations can address their respective limits.The passage identifies generalization beyond literal record-and-replay macros as a central PBD challenge.Source: The Multi-Modal Approach · p. 3
- PUMICE uses natural-language conversation as its main modality and demonstrations to ground concepts, values, and procedures after conversational explanation.It extends prior multimodal PBD work to learning concepts that can generalize to new task domains.Source: The Multi-Modal Approach · p. 4
Formative Study
A formative study examined how people instruct agents about conditional tasks in mobile apps, including whether app screenshots affect their descriptions. The findings document vague concepts, omitted alternatives, and expectations that agents can reason with common sense.
Formative Study
- The formative study involved 58 participants, including 41 non-programmers, who described conditional tasks in mobile-app scenarios.Participants were assigned one of nine tasks and instructed using graphical task descriptions with limited text.Source: Formative Study · p. 4
- Participants were randomly assigned to describe tasks with or without relevant app screenshots, and were also asked to propose a conditional task of their own.Two independent coders analyzed responses; agreement was κ = 0.87, and 19% were excluded for quality-control reasons.Source: Formative Study · p. 4
App GUI Grounding Reduces Unclear Concept Usage
- In the control group without app screenshots, 33% used ambiguous or vague concepts such as “daytime” to specify a condition.This occurred despite participants having seen example instructions with clearly defined conditions.Source: App GUI Grounding Reduces Unclear… · p. 4
- With screenshots showing task-relevant app information, 9% used ambiguous or vague concepts, versus 33% in the control group (p < 0.05).The authors suggest concrete app values can help users ground concepts in their descriptions.Source: App GUI Grounding Reduces Unclear… · p. 4
Unmet User Expectation of Common Sense Reasoning
- Participants often expected agents to understand common-sense concepts such as “weekend” and “sunset time,” including how to determine their values.The paper identifies grounding new concepts in app contents and building knowledge through user interaction as a feasible workaround.Source: Unmet User Expectation of Common Sense… · p. 4
Frequent Omission of Else Statements
- Else statements were omitted in 18% of descriptions when expected and 33% when a person might infer an alternative situation.The authors suggest users may omit what they expect an agent to infer, and recommend proactively asking about alternatives when appropriate.Source: Frequent Omission of Else Statements · pp. 4–5
PUMICE
PUMICE recursively resolves unclear concepts in conditional instructions through conversation and demonstrations, then stores learned concepts and procedures for reuse. Its design supports context-sensitive generalization, error recovery, and GUI-based value queries; its implementation combines semantic parsing with Android app demonstration recording.
Example Scenario
- In the example, PUMICE identifies an unknown condition and action, then learns a reusable temperature query from a user demonstration.It recognizes the condition as Boolean and asks how to determine the temperature before the user demonstrates in a weather app.Source: Example Scenario · p. 5
- A Starbucks demonstration teaches a generalized ordering procedure with the drink as a parameter, allowing a known item to be used for the else action.PUMICE can recognize “Hot Latte” as an available parameter when the user specifies what to do if it is not hot.Source: Example Scenario · p. 5
PUMICE
- PUMICE recursively resolves typed unknown concepts and stores fully resolved concepts in a persistent knowledge graph.It prompts users to explain Boolean concepts or value concepts as needed.Source: PUMICE · p. 5, Support for Concept Learning · p. 6
Design Features
- Users can explain Boolean concepts verbally or as expressions, and demonstrate GUI queries for concrete value concepts that PUMICE stores for reuse.PUMICE infers a target value’s type where possible, highlights candidate GUI items, and records navigation and value extraction as a query.Source: Support for Concept Learning · p. 6
- Boolean concepts retain the same operation while allowing different arguments, and value concepts with the same name can use context-specific queries.For “hot,” the threshold or the value query can differ across contexts.Source: Concept Generalization and Reuse · p. 7
- For parsing errors, PUMICE can ask users to disambiguate uncertain expression components and provides undo to restore an earlier conversational state.It compares candidate parses and confidence scores to identify components for clarification.Source: Error Recovery and Backtracking · p. 7
System Implementation
- PUMICE parses utterances into a functional DSL using SEMPRE’s Floating Parser and typed resolve() functions for unknown concepts or procedures.The parser adds relevant utterances and GUI text labels to its lexicon and was trained on 905 examples.Source: Semantic Parsing · p. 8
- SUGILITE records and replays Android app actions, but PUMICE may have problems with web apps and special graphic engines and does not record gestural or sensory inputs.The framework uses Android Accessibility APIs and supports parameterized action sequences.Source: Demonstration Recording and Replaying · p. 8
User Study
The lab study tested whether users could teach PUMICE conditional smartphone automations using concepts and procedures across apps. Ten users completed four tasks, and post-study responses and interviews assessed usability; the authors note tutorial guidance may have contributed to participants’ success.
User Study
- The task-based evaluation tested usability without a control condition because the authors found no feasible alternative for users with little programming experience.Each participant created four automations with required conditionals and new concepts.Source: User Study · p. 8
Participants
- The 10 participants included five women and five men, aged 19–35; six were students, and four had technical, administrative, or managerial jobs.One had never programmed, five had used end-user programming tools, and four had programming experience.Source: Participants · p. 9
Procedure
- Each session began with a five-minute tutorial, then used a Pixel 2 with PUMICE and installed apps; participants tried four randomized tasks until satisfied, followed by a survey and interview.Researchers checked the resulting scripts for correctness.Source: Procedure · p. 9
Tasks
- The four tasks exercised concept learning, value-query and procedure demonstrations, generalization, and else handling across familiar apps and controllable conditions.Task 1 required generalizing “hot” to coffee ordering and demonstrating iced coffee; the else action could be generalized from that demonstration.Source: Tasks · p. 9
Results
- All 10 participants completed all four tasks; programmers averaged 22.12 minutes (SD=2.40) versus non-programmers’ 23.06 minutes (SD=1.57), a nonsignificant difference.Figure 5 shows average completion times for each task and compares the two programming-experience groups.Source: Figure 5 · p. 10, Results · p. 10
- Participants resolved speech-recognition and parsing errors through undoing or rephrasing; the authors caution that tutorial examples and app-grounded concept guidance may have contributed to success.One participant corrected a mistaken Boolean comparison through PUMICE’s mixed-initiative mechanism.Source: Results · p. 10
- On a 7-point scale, PUMICE averaged 6.2 for ease of use, 6.1 for naturalness, and 6.9 for usefulness.Participants were generally satisfied with their experience using PUMICE.Source: Results · p. 10
Discussion
- Interviewees praised PUMICE’s naturalness and demonstration support, and found interactive concept resolution useful because they need not define everything upfront.Non-programmers were particularly impressed by the multimodal interface.Source: Discussion · p. 10
Limitations and Future Work
The authors identify limits in PUMICE’s semantic understanding, conditional expressiveness, sharing, and execution model, and plan further field evaluation. These boundaries constrain which concepts, rules, users, and settings the current prototype can support.
Limitations and Future Work
- PUMICE lacks semantic understanding of task information, so it cannot handle implicit parameters or relations between concepts.The parser also does not process references, synonyms, antonyms, or implicit conjunctions and disjunctions.Source: Limitations and Future Work · pp. 10–11
- Its conditionals support only single basic comparisons, not logical or arithmetic operations or counting GUI elements.The authors plan to explore interfaces for these features, noting that users’ language for “and,” “or,” and computation can be ambiguous.Source: Limitations and Future Work · p. 11
- Generalized procedures and concepts are stored locally and limited to one user; the prototype also does not distinguish personal preferences from generalizable knowledge.Sharing knowledge across users would require mechanisms to preserve privacy.Source: Limitations and Future Work · p. 11
- The prototype supports immediate task performance rather than fully automated rules, and its foreground-only execution cannot continuously monitor GUI values in the background.The authors plan to investigate virtual machines for background execution.Source: Limitations and Future Work · p. 11
- The usability evidence comes from an in-lab study; an open-ended field study is planned to examine real-life tasks, modality choices, and usefulness.The authors seek to understand how PUMICE is used in realistic contexts.Source: Limitations and Future Work · p. 11
Conclusion
PUMICE learns concepts and conditionals through conversational instructions and demonstrations, combining interaction modalities to address unclear concepts. Users can ground abstract task conditions in smaller concepts and third-party mobile-app content.
Conclusion
- PUMICE learns concepts and conditionals from conversational natural-language instructions and demonstrations, addressing unknown, ambiguous, or vague concepts common in users’ verbal commands.The paper presents multimodal interaction as its approach to learning these concepts.Source: Conclusion · p. 11
- Users can explain abstract conditions through smaller concepts and ground them with third-party mobile-app demonstrations or GUI references.The paper presents combining conversational and demonstrational interfaces as a way to create natural end-user development experiences.Source: Conclusion · p. 11