Source-linked AI summary
Efficient Model Learning for Human-Robot Collaborative Tasks
Stefanos Nikolaidis, Keren Gu, Ramya Ramakrishnan, Julie Shah
TL;DR
The paper addresses how robots can learn personalized human teammate models for collaborative tasks without manual intervention. It clusters joint-action demonstrations, learns type-specific rewards through inverse reinforcement learning, and embeds them in an MOMDP with partially observable human type. The resulting policies infer new users offline or online, remain robust to deviations from demonstrations, and achieve performance comparable to a hand-coded expert model.
Problem
Robots need a principled way to learn personalized human behavior models and integrate them into collaborative-task decision making without relying on manual expert teaching.
Method
The framework clusters team-action demonstrations into human types, learns a reward function for each type through inverse reinforcement learning, and uses these rewards in an MOMDP.
Results
The learned policies were robust to increasing deviations from demonstrated human behavior and performed comparably to policies based on a domain-expert hand-coded model.
Takeaways & Limitations
Human-type models can be learned efficiently and integrated into decision making to produce policies aligned with a human teammate’s personalized style.
Takeaways & Limitations
The framework assumes that high-level human team strategies are limited in number, allowing them to be represented by a constrained set of human types.
Abstract
from arXiv · showhide
We present a framework for learning human user models from joint-action demonstrations that enables the robot to compute a robust policy for a collaborative task with a human. The learning takes place completely automatically, without any human intervention. First, we describe the clustering of demonstrated action sequences into different human types using an unsupervised learning algorithm. These demonstrated sequences are also used by the robot to learn a reward function that is representative for each type, through the employment of an inverse reinforcement learning algorithm. The learned model is then used as part of a Mixed Observability Markov Decision Process formulation, wherein the human type is a partially observable variable. With this framework, we can infer, either offline or online, the human type of a new user that was not included in the training set, and can compute a policy for the robot that will be aligned to the preference of this new user and will be robust to deviations of the human actions from prior demonstrations. Finally we validate the approach using data collected in human subject experiments, and conduct proof-of-concept demonstrations in which a person performs a collaborative task with a small industrial robot.
INTRODUCTION
The paper introduces an automatic framework that learns human user types and type-specific rewards from team demonstrations, then uses a MOMDP to compute robust, preference-aligned robot policies. It addresses the need to model personalized human behavior without requiring manual task-specific teaching.
- Motivation and contribution: The framework automatically learns human user models from joint-action demonstrations and computes robust policies for collaborative tasks.It assumes demonstrations from human teams working on the task and compares robustness against previous model-learning algorithms.
- Motivation and contribution: Demonstrated action sequences are clustered into a limited set of dominant human types using unsupervised learning.The approach relies on the observation that varied human preferences often produce relatively few high-level team strategies.
- Motivation and contribution: The MOMDP represents human type as a partially observable variable and makes the robot’s reward depend on that type.This models uncertainty about the teammate’s preference for the robot’s task-related actions.
- Motivation and contribution: Each human type receives a reward function learned from its clustered demonstrations through inverse reinforcement learning.This avoids manually specifying a reward function for every possible human preference.
- Validation and implications: The learned framework estimates a new teammate’s type offline or online and computes policies aligned with that preference while tolerating action deviations.Human-subject experiments and robot demonstrations were used for validation.
- Relation to prior work: The approach differs from prior work by learning the MOMDP reward structure rather than assuming a known human model or reward.Related approaches also include interaction-based model learning, pursuit-game teammate models, and game-AI intention inference.
METHOD
The method has a training stage that clusters demonstrated team-action sequences into dominant human types and a deployment stage that infers a new teammate’s type. The robot then selects actions using an MOMDP policy conditioned on its current belief about that type.
- Training stage: The framework first preprocesses training data and clusters demonstrated human-team action sequences into dominant human types.The clustering identifies the number of types used to represent personalized collaborative styles.
- Policy execution: The robot executes the MOMDP policy at each time step using its current belief about the human type.The flowchart summarizes the framework’s two-stage progression from training data to personalized execution.
- Inference stage: For a new teammate, the robot computes a belief over human types from a few demonstrated human-and-robot action sequences.The type can instead be estimated online when the human’s actions are informative about preferences for the robot’s actions.
CLUSTERING OF HUMAN TYPES
The framework clusters joint human–robot action sequences into a limited set of human types, then uses those clusters to construct type-specific models for MOMDP planning.
- Human-type representation: Human preferences are represented as a limited set of high-level types inferred from demonstrated alternating human–robot action sequences.Each sequence is assigned to a cluster representing a human preference pattern.
- EM clustering: A hard EM procedure represents each cluster with a transition matrix over robot and human actions.The matrix encodes how each agent acts based on the preceding action of the other agent.
- EM clustering: EM alternates sequence assignment and transition-matrix updates until the assignments stabilize.The E-step assigns sequences using log-likelihood, while the M-step counts transitions within each cluster.
- Model selection: The ideal number of clusters is selected with BIC by running EM across candidate values of k.The stated candidate range is k = 2 to 10, with multiple model evaluations used for selection.
- MOMDP integration: The learned clusters are mapped to human-type values in a MOMDP, where each type receives a learned reward function for robot planning.The resulting policy can reason over uncertainty about the human type.
MOMDP Formulation
The MOMDP separates observable task progress from the partially observable human type, allowing the robot to plan actions under uncertainty about the collaborator’s preferences.
- State representation: The MOMDP state combines the observable task-step variable x with the unobservable human-type variable y.The task-step is assumed fully observable, while the human type is inferred from observations.
- Dynamics and actions: Robot actions form a finite set of discrete task-level actions, and the observable-state transition model predicts the next task step.The transition function Tx maps the current state and robot action to a distribution over next observable states.
- Dynamics and actions: The partially observable transition model predicts the next human type conditioned on the current state, robot action, and next task step.This is represented by the function Ty.
- Reward and planning: The reward depends on the task step, human type, and robot action, capturing the type-specific value of robot behavior.The robot selects actions using this reward structure while accounting for uncertainty over the human type.
- Observations: Observations consist of actions taken by both the human and the robot, and are used by the robot to update its information about the hidden type.The observation function assigns probabilities to possible observations given the state and robot action.
Belief-State Estimation
The framework performs belief-state estimation for the hidden human type using the MOMDP’s transition and observation models.
- Belief update: The robot updates its belief over human types using the MOMDP belief-update procedure.The update is based on the current belief together with the model’s transition terms.
Inverse Reinforcement Learning
The framework learns a reward function for each human type from demonstrations assigned to that type, avoiding manual reward specification. It applies inverse reinforcement learning by matching demonstrated and policy feature expectations, then aggregates later-iteration weights into the output reward.
- Reward Learning: Manual reward specification is tedious and time-consuming, so the framework learns a reward function for each human type from clustered demonstrations.Each cluster’s demonstrations are used to learn the reward associated with that type.
- IRL Formulation: For a fixed human type, the MOMDP is reduced to an MDP, enabling inverse reinforcement learning from demonstrated state-action sequences.The method assumes the human type remains constant within the demonstrated sequences.
- IRL Procedure: The IRL procedure iteratively generates policies whose feature expectations approach those of demonstrations from humans of the same type.It begins with a random policy and terminates when the feature-expectation distance falls below ϵ.
- IRL Procedure: The algorithm computes reward weights through convex quadratic programming and reinforcement-learning updates of policies for the resulting reward.The iterative process uses Monte Carlo estimates of policy feature expectations.
- Reward Output: The final reward function averages weight values from the second half of the iterations, excluding lower-quality initial policies and weights.The IRL output includes a list of policies, their weights, and feature counts.
Policy Computation
The robot computes a policy with a Mixed Observability Markov Decision Process that accounts for uncertainty about the human type. The implementation uses SARSOP, a point-based approximation solver, because exact POMDP solutions are computationally expensive.
- Policy Computation: The MOMDP policy accounts for uncertainty over the human type while maximizing the robot’s expected total reward.MOMDPs are structured variants of POMDPs with observable and partially observable variables.
- Policy Computation: SARSOP is used as a point-based approximation method because exact POMDP solution is computationally expensive.Point-based methods update selected belief points to accelerate planning.
EVALUATION
The evaluation applies the framework to place-and-drill and hand-finishing teamwork, with the place-and-drill experiment using demonstrations from human subjects. In that task, learned human types and reward functions are passed to a MOMDP to compute robot actions for a held-out subject.
- Evaluation: The framework is evaluated on two human-robot teamwork applications: place-and-drill and hand-finishing.The supplied figure depicts task execution by a human-robot team on the place-and-drill task.
- Place-and-Drill Task: In place-and-drill, 18 subjects demonstrated a shared-location task where humans placed screws and the robot drilled each screw.Training included role switching so subjects could demonstrate preferred robot drilling actions.
- Evaluation Protocol: Leave-one-out evaluation removed one subject for testing and trained the clustering algorithm on the remaining 17 subjects.Across cross-validation iterations, subjects were clustered into safe and efficient types.
- MOMDP Inputs: The safe and efficient types define alternative drilling preferences, and their learned reward functions are supplied to the MOMDP as the two possible human-type values.The observable state is the workbench configuration, and robot actions include drilling and no-op actions.
- Policy Evaluation: For each held-out subject, three demonstrations produced an initial type belief, which SARSOP used with the learned reward functions to compute a robot policy for task execution.The subject then executed the task with the actual robot in the predefined roles.
Performance of Human Type Clustering
Human action sequences were clustered into safe and efficient types using place-and-drill demonstrations. The clustering agreed with expert labels at an average accuracy of 96.5%, while robot-as-human action orders matched human orders 88% of the time.
- Data: 54 demonstrated sequences from 18 participants were clustered, using the role-reversal phase that revealed preferences for robot actions.Each participant contributed three sequences.
- Data Validation: Robot-as-human action orders matched the human action orders 88% of the time, supporting their use for clustering evaluation.The orders were similar even though the robot did not reproduce human actions exactly.
- Validation: Leave-one-out cross-validation assigned each participant a final type by taking a weighted average over that participant’s three sequence assignments.Weights came from likelihoods calculated with the Cluster-Transition-Matrices EM algorithm.
- Results: 96.5% average classification accuracy was obtained against manual expert labels.Both the algorithm and expert identified safe and efficient types.
- Results: Safe participants preferred waiting until all screws were placed before drilling, whereas efficient participants preferred alternating placement and drilling to finish quickly.The reported accuracy was attributed to clear separation in distance between the two types.
Robustness of Computed Policy
The automatically learned MOMDP policy maintained accumulated reward as human actions increasingly deviated from demonstrations, unlike human-robot cross-training, while matching a hand-coded model under comparable evaluation.
- Increasing deviations reduced the cross-training policy’s performance, whereas the proposed MOMDP policy was not affected.Deviations were simulated by replacing demonstrated human placement actions with random actions at probability ϵ.
- With no deviation from demonstrated placement actions, the cross-training and proposed policies performed similarly.
- The automatically generated MOMDP model achieved comparable performance to a domain expert’s hand-coded model.Accumulated reward was averaged across cross-validation iterations.
- Figure 5 compares accumulated reward across 18 cross-validation iterations while varying the probability of random human actions.Each subject contributed 100 simulated task-execution iterations.
Online Estimation of Human Type
The framework estimates a new human’s preference type online within a MOMDP and uses that belief to guide robot actions in a collaborative hand-finishing task.
- Six subjects’ demonstrations produced two clusters corresponding to refinishing the left or right side first.These preferences formed the unobserved human-type states in a 2000-state MOMDP.
- The MOMDP estimates human type from observations of hand position while the robot controls the box’s position and tilt.The observation function models hand position with a discretized 2D Gaussian conditioned on refinishing preference.
- During execution, the robot updated its belief as the person moved and waited for more information before moving the box.The belief initially favored presenting the right side first, then was updated after the person moved toward the robot and slightly left.
- The framework supports inferring a new human’s type either offline from demonstrations or online during task execution.
- Evaluation found robust performance under increasing deviations and performance comparable to a MOMDP using a domain expert’s hand-coded model.