Source-linked AI summary
Longitudinal Robot Learning from Demonstration with Care Providers in a Home Environment
Nina Moorman, Julianna Schalkwyk, Vriksha Srihari, Qingyu Xiao, Kamel Alrashedy, Hongseok Jeong, Kiersten Lange, Matthew B. Luebbers, Matthew Gombolay
TL;DR
The paper addresses limited evidence about non-experts teaching robots through LfD outside controlled laboratories and without live robotics experts. It evaluates pre-training and adaptive feedback with care providers teaching assistive tasks in a home over multiple visits, while proposing an open dataset and design guidance.
Problem
Most non-expert LfD usability evaluations occur in controlled laboratories with robotics experimenters, leaving home-based teaching without live expert feedback insufficiently examined.
Method
The study uses a multi-visit home evaluation in which care providers teach assistive tasks with pre-training, adaptive feedback, and an interactive kinesthetic demonstration interface.
Results
The work identifies non-expert users’ barriers when teaching robots in a home environment and proposes open sourcing the resulting multi-visit assistive-task dataset.
Takeaways & Limitations
The dataset and planned design guidelines support studying and benchmarking interactive robot learning with care providers in realistic assistive settings.
Abstract
from arXiv · showhide
Learning from demonstration (LfD) methods enable non-expert end users to teach robots novel skills without explicit programming. However most evaluations of the usability of LfD with non-experts has been conducted in controlled laboratory environments with a robotics experimenter present. In this work we identify non-expert end users' key barriers when teaching robots via demonstration without live robotics expert feedback in a home environment. In our human subjects experiment we support the non-expert end users through two forms of demonstrator guidance developed in prior work: pre-training and adaptive feedback. Towards the ecological validity of the evaluation, we conduct this experimentation over multiple visits, with a population of care providers. Finally, we propose to open source the resulting LfD dataset of care providers teaching a robot assistive tasks over multiple visits to a home environment.
I. INTRODUCTION
The paper examines barriers faced by non-experts teaching robots through LfD without live expert feedback, emphasizing realistic, multi-visit evaluation with care providers in a home environment.
- Real-world robot learning requires adaptation to unstructured, dynamic human environments, but collecting large-scale in-home data is costly, time-consuming, and privacy-sensitive.
- The study investigates remaining challenges for non-experts teaching robots through LfD across multiple visits and tasks.Participants define and demonstrate custom unimanual and bimanual manipulation tasks in a realistic home environment.
- Participants receive automated training and feedback to support teaching robots assistive household tasks without live robotics expert feedback.
- The experiment is designed to characterize opportunities for deploying LfD with non-experts in a home environment using pre-training and adaptive feedback.
- The resulting multi-visit dataset of non-experts teaching assistive tasks is intended for open release.
II. DEMONSTRATION SYSTEM
The demonstration system combines perception, probabilistic skill learning, kinesthetic task construction, pre-training, and adaptive feedback to support non-experts teaching robots.
- An RGB-D stream is processed to detect objects, generate segmentation masks, reconstruct 3D point clouds, and estimate tracked-object 6D poses.YOLOE produces masks, back-projection produces object-specific point clouds, and ICP registers them against predefined shapes.
- Cartesian ProMPs learn end-effector and tracked-object pose trajectories from limited demonstrations while supporting fast training and generalization across object poses.The paper contrasts this choice with Behavior Cloning and Vision-Language Action models, which may struggle with generalization or user-specific adaptation.
- The interactive interface lets participants decompose tasks into low-level skills, demonstrate each skill kinesthetically, and compose skill recipes.
- Pre-Training: Pre-training shows participants expert task decompositions and demonstrations after their initial attempts, enabling improvement on novel domains without live expert feedback.
- Adaptive Feedback: Visit domains balance task complexity and unimanual-to-bimanual task distribution while using Tupperware stacking as kinesthetic onboarding.
- Adaptive Feedback: FM feedback assesses whether planned task decompositions can generalize downstream and asks clarification questions to improve novice domain descriptions.
- Adaptive Feedback: Real Robot Replay and Augmented Reality feedback let users observe learned-policy performance, with AR avoiding time-consuming or dangerous environmental interactions.
III. DATA COLLECTION & HUMAN EVALUATION
The study evaluates care providers teaching assistive tasks to robot arms across three visits in an authentic home, comparing pre-training plus adaptive feedback with adaptive feedback alone.
- The evaluation recruits formal and informal care providers and takes place in the Georgia Tech Aware Home.
- Participants teach tasks to two JACO 2 arms over three visits, progressing from pre-trained domains to additional domains and then custom tasks.
- The PT+AF condition receives pre-training in visit 1 and adaptive feedback in visits 2 and 3, whereas the AF condition receives adaptive feedback only in visits 2 and 3.
- Research Questions: RQ1 compares conditions on task completion percentage using a predefined rubric after each visit.
- Research Questions: RQ2 measures alignment as the absolute difference between task performance and normalized perceived performance.
- Research Questions: RQ3 examines usability, acceptance, learned trust, and workload using mixed-effects linear regressions controlling for time-on-task, demonstrations, and visit count.
IV. OUTCOMES
The work plans to release a richly structured, multi-visit care-provider dataset and report human-factors findings alongside deployment lessons and design guidelines for home-based assistive LfD.
- The dataset will supplement existing robot-skill datasets with realistic assistive tasks trained in situ by care providers over multiple visits.
- Its structure includes participant, domain, task, and skill information, with natural-language descriptions and object point clouds.
- The released dataset is intended to enable standardized benchmarking for interactive robot-learning algorithm development and evaluation.
- Demonstrations include end-effector and joint trajectories, participant audio, and egocentric and exocentric video footage.
- The authors plan to report human-factors findings from three research questions and lessons learned from deploying LfD systems in home environments.
- They also plan design guidelines for assistive-task deployment, care-provider-focused systems, and multi-visit demonstration-data collection.