Source-linked AI summary
Learning Preferences for Manipulation Tasks from Online Coactive Feedback
Ashesh Jain, Shikhar Sharma, Thorsten Joachims, Ashutosh Saxena
TL;DR
High-DoF robots in homes and grocery stores may produce undesirable motions because they lack an understanding of user preferences, which can depend on surrounding objects and people. The paper uses coactive online learning from incremental trajectory improvements and reports that users can train robots within minutes across household and grocery tasks.
Problem
High-DoF robots in unstructured settings may produce undesirable motions because they lack an understanding of user preferences, which can depend on surrounding objects and human interactions.
Source: §I · p. 1Method
The robot proposes a trajectory, then updates its preference model from a user's slightly improved trajectory without requiring an optimal demonstration.
Source: §I · p. 1, A. Robot learning setup · p. 3Results
Across household and grocery tasks, TPP learned good trajectories quickly, generalized to unseen tasks, and outperformed baselines; users trained robots within a few minutes with incremental feedback.
Source: §I · p. 2, B. Results and Discussion · p. 10, §VII · p. 13Takeaways & Limitations
The approach addresses preferences beyond geometric constraints, including context involving objects and people in household and grocery settings.
Source: §I · p. 2, §VII · p. 13
Abstract
from arXiv · showhide
We consider the problem of learning preferences over trajectories for mobile manipulators such as personal robots and assembly line robots. The preferences we learn are more intricate than simple geometric constraints on trajectories; they are rather governed by the surrounding context of various objects and human interactions in the environment. We propose a coactive online learning framework for teaching preferences in contextually rich environments. The key novelty of our approach lies in the type of feedback expected from the user: the human user does not need to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system. We argue that this coactive preference feedback can be more easily elicited than demonstrations of optimal trajectories. Nevertheless, theoretical regret bounds of our algorithm match the asymptotic rates of optimal trajectory algorithms. We implement our algorithm on two high degree-of-freedom robots, PR2 and Baxter, and present three intuitive mechanisms for providing such incremental feedback. In our experimental evaluation we consider two context rich settings -- household chores and grocery store checkout -- and show that users are able to train the robot with just a few feedbacks (taking only a few minutes).\footnote{Parts of this work has been published at NIPS and ISRR conferences~\citep{Jain13,Jain13b}. This journal submission presents a consistent full paper, and also includes the proof of regret bounds, more details of the robotic system, and a thorough related work.}
I. INTRODUCTION
High-DoF robots need trajectories that satisfy contextual user preferences, which can be difficult to encode or demonstrate in advance. The paper learns these preferences from incremental feedback and evaluates the approach across household and grocery tasks.
I. INTRODUCTION
- High-DoF robots need trajectories that satisfy user preferences as well as geometric constraints, and contextual preferences can be hard to anticipate and encode manually.Examples include keeping water upright and moving heavy items around fragile objects.Source: §I · p. 1
- The approach learns from trajectories users incrementally improve, without requiring them to demonstrate optimal trajectories, and provides regret bounds matching optimal-demonstration algorithms’ asymptotic rates.The framework updates an online model of user preferences from these improvements.Source: §I · p. 1
- Across 35 household and 16 grocery-checkout tasks, the system learned good trajectories quickly and generalized well to tasks it had not seen.The evaluation included batch and robotic experiments with users providing preferences.Source: §I · p. 2
- The paper’s contributions include feedback mechanisms for improving trajectories and expressive features for capturing contextual information beyond geometric criteria.The approach is implemented on PR2 and Baxter and supported by a user study.Source: §I · p. 2
II. RELATED WORK
- Unlike learning from demonstration, which asks for optimal trajectories, this approach learns preferences from suboptimal suggestions and models contextual object and human interactions.The paper describes its trajectory features as incorporating robot configurations, object-object relations, and temporal behavior.Source: §II · pp. 2–3
III. COACTIVE LEARNING WITH INCREMENTAL FEEDBACK
The robot predicts a trajectory, receives an incremental user improvement, and updates its preference score online. Users can provide that improvement by re-ranking trajectories, correcting a waypoint in zero-G mode, or editing it through an interactive interface.
A. Robot learning setup
- At each step, the robot predicts a trajectory using its current score, observes an improved trajectory, and updates the score to better match user preferences.PR2 and Baxter each have two 7-DoF arms; proposed trajectories are visualized in simulation before users choose feedback.Source: A. Robot learning setup · p. 3
B. Feedback mechanisms
- In re-ranking, users inspect trajectories in predicted-score order and move a lower-ranked trajectory above the current top choice when it is better.The described examples include selecting the third-ranked trajectory for a knife task and the second-ranked trajectory for an egg-carton task.Source: B. Feedback mechanisms · p. 4, Figure 3 · p. 4
- In zero-G mode, users physically steer the arm to correct a waypoint; this can bootstrap learning, escape correctly ordered but poor rankings, or fix minor errors.The mode is activated on Baxter when a user holds its wrist.Source: B. Feedback mechanisms · p. 4
- For robots without zero-G capability, an interactive Rviz-ROS interface lets users improve trajectories by correcting waypoints.The paper describes this interface as serving the same purpose as zero-G feedback.Source: B. Feedback mechanisms · p. 4, Figure 4 · p. 4
- Across all three mechanisms, users provide trajectories that are slightly improved in expectation rather than revealing the optimal trajectory.The algorithm is intended to learn from such non-expert feedback.Source: B. Feedback mechanisms · p. 4
IV. LEARNING AND FEEDBACK MODEL
The framework models trajectory quality as a user-valued score conditioned on task context, then learns that score from preference feedback rather than requiring optimal demonstrations. Its scoring function combines features for object-object interactions with features for the manipulated object and its environment.
IV. LEARNING AND FEEDBACK MODEL
- The user’s unobserved scoring function ranks trajectories for a context describing the environment, objects, and other task-relevant inputs.The robot learns an approximate scoring function; the user need only provide preferences, not cardinal valuations.Source: §IV · pp. 4–5
- At each iteration, the robot ranks sampled trajectories, and the user either accepts the top trajectory or provides an improved one for the robot’s next update.The feedback indicates that the improved trajectory has a higher user score than the proposed trajectory.Source: §IV · p. 5
- Regret compares the robot’s trajectory with the user-scoring optimum over its lifetime, while non-expert users provide better-than-proposed trajectories rather than optimal demonstrations.The robot’s regret is expressed using the true scoring function, even though that function is unobserved.Source: §IV · p. 5
V. LEARNING ALGORITHM
- The learned score is linear in trajectory features and decomposes into an object-interaction term and an object-environment term.The two terms have separate weights and feature vectors.Source: §V · p. 5
A. Features Describing Object-Object Interactions
Object-object interaction features connect trajectory waypoints to relevant environmental objects, represent object properties with binary attributes, and score interactions using proximity and vertical-position cues.
A. Features Describing Object-Object Interactions
- A waypoint connects to an environmental object when their minimum collision distance falls below a threshold or the object lies below the manipulated object.The connection identifies which objects may affect a trajectory waypoint.Source: A. Features Describing Object-Object… · p. 5
- Each object is represented by binary labels for properties such as being heavy, fragile, sharp, hot, liquid, or electronic.For example, the passage gives distinct attribute vectors for a laptop and a glass table.Source: A. Features Describing Object-Object… · pp. 5–6
- For each waypoint-object edge, four features encode minimum collision-distance projections along x, y, and z, plus whether the object lies vertically below the manipulated object.Source: A. Features Describing Object-Object… · p. 6
- The interaction score is defined over the graph, with weights indexed by pairs of object properties.The overall object-interaction weight vector concatenates the property-pair weight vectors.Source: A. Features Describing Object-Object… · p. 6
B. Trajectory Features
The trajectory features capture arm configuration, object orientation and motion over time, and the manipulated object’s distances from surrounding surfaces. Trajectories are ranked by score, with sampling used to generate candidates for high-degree-of-freedom manipulators.
1) Robot Arm Configurations:
- The arm-configuration features track elbow and wrist coordinates in three trajectory segments, yielding 27 features that include extrema and elbow locations at wrist extrema.The features are motivated by preferences among arm configurations that can reach the same wrist position, including avoiding configurations that may cause joint locks.Source: 1) Robot Arm Configurations · p. 6
2) Orientation and Temporal Behaviour of the Object to be Manipulated:
- Object features capture orientation requirements and motion variation across three trajectory segments, including orientation deviation and frequency-based measures of oscillation.The representation contains 28 features, including the object’s maximum deviation over the full trajectory; Fig. 6 contrasts good and bad mug motions and their z-direction spectrograms.Source: 2) Orientation and Temporal Behaviour… · p. 6, Figure 6 · p. 6
3) Object-Environment Interactions:
- Object-environment features track vertical and horizontal clearances over time, distinguishing a trajectory that grazes a table edge from one that rises over it smoothly.The representation uses 20 features, including distances to nearby surfaces, the table, and the goal, plus features from a time-frequency spectrogram.Source: 3) Object-Environment Interactions · p. 6
C. Computing Trajectory Rankings
- For high-degree-of-freedom manipulators, the authors sample trajectories with RRT and add diversity methods to avoid returning many similar candidates.They rank discrete candidates by trajectory score; sampling may use obstacles or randomized goal-sample bias, among other methods.Source: C. Computing Trajectory Rankings · p. 7
D. Learning the Scoring Function
The Trajectory Preference Perceptron learns score weights from feature differences between a proposed trajectory and a user-improved one. Its regret guarantee allows imperfect, non-optimal feedback under an expected-improvement condition.
D. Learning the Scoring Function
- Trajectory Preference Perceptron updates each feature-weight vector in the direction of the difference between the user-improved and currently top-ranked trajectories.The update does not require an optimal demonstration.Source: D. Learning the Scoring Function · p. 7
- The regret guarantee applies when feedback is expected to improve the proposed trajectory by a fraction α of the maximum possible score range, with slack ξ_t accounting for violations.The condition is in expectation rather than deterministic, allowing noisy or imperfect feedback.Source: D. Learning the Scoring Function · p. 7
- The feedback criterion requires only an expected improvement over the robot’s proposal, whose quality improves over iterations in practice.Source: D. Learning the Scoring Function · p. 7
A. Experimental Setup
The evaluation covers household and grocery-store manipulation with PR2 and Baxter, across tasks involving object motion, surrounding objects, and human safety. It compares learning settings and feedback methods using expert trajectory ratings and ranking metrics.
A. Experimental Setup
- The study evaluates 35 household tasks on PR2 and 16 grocery checkout tasks on Baxter, testing generalization across changes in objects and environments.Trajectory quality is assessed after grasping, while the robot moves the item to complete the task.Source: A. Experimental Setup · p. 8
- Activities span manipulation-centric tasks such as pouring, environment-centric interactions such as moving water near electronics, and human-centric handling of sharp or hot objects.Examples include pouring and inserting, moving fragile items or liquids near electronics, and handling a knife near a person.Source: A. Experimental Setup · p. 8, Figure 8 · p. 9
- The experiments study generalization to unseen tasks, learning without demonstrations, and the effectiveness of different feedback types.Algorithms are evaluated from scratch and after pretraining on similar tasks before adapting to a new task.Source: A. Experimental Setup · p. 8
- Baselines include task-independent BiRRT planning, manually encoded preferences, and MMP-online, which treats feedback as optimal demonstrations and retrains an SSVM.MMP-online uses a hindsight-selected regularization parameter, giving it an acknowledged advantage in the comparison.Source: A. Experimental Setup · p. 8
- Expert raters scored 1,300 grocery and 2,100 household trajectories on a five-point Likert scale; these ratings were reserved for evaluation, not algorithm input.The highest score is 5.Source: A. Experimental Setup · p. 8
- Performance is measured by nDCG@1 and nDCG@3, which suit autonomous execution and human-supervised ranking, respectively.nDCG uses trajectory Likert scores and normalizes against the ideal ranking by descending score.Source: A. Experimental Setup · p. 9
B. Results and Discussion
TPP performs better than the compared baselines across household and grocery activities. Pretraining improves initial generalization, while more informative feedback reaches a given ranking quality with fewer iterations.
B. Results and Discussion
- Pretrained TPP starts with higher nDCG@3 than untrained TPP under object, environment, or combined changes, but both converge to identical performance as feedback accumulates.Generalization is harder when both the object and environment change than when only one changes.Source: B. Results and Discussion · pp. 9–10, Figure 9 · p. 10
- MMP-online performs much worse than TPP in both settings; treating feedback as optimal demonstrations accumulates contradictory or suboptimal examples.The paper reports better MMP-online generalization in grocery than household tasks, while noting its sensitivity to suboptimal demonstrations.Source: B. Results and Discussion · p. 10
- 0.44 nDCG@3 is the manual baseline’s average across household activities; TPP performs better than other baselines in both household and grocery settings.The comparison reports nDCG values averaged over 20 feedback iterations in the untrained setting.Source: B. Results and Discussion · p. 10
- More informative feedback reaches a given nDCG@1 with fewer iterations: click-one-to-replace-top and click-one-from-5 outperform approximate-argmax.Approximate-argmax improves slowly, although all three feedback types are α-informative and eventually learn preferences.Source: Figure 10 · p. 10, B. Results and Discussion · pp. 10–11
C. Comparison with fully-supervised algorithms
On new tasks, TPP improves over the fully supervised Oracle-svm after an average of five feedback iterations. The user study also shows that participants trained PR2 and Baxter with a small number of feedbacks, in about 5.5 minutes per grocery task on average.
C. Comparison with fully-supervised algorithms
- After an average of 5 feedback iterations, TPP improves over Oracle-svm on new tasks, while Oracle-svm remains unchanged without expert labels.Oracle-svm initially performs better without feedback on new tasks.Source: C. Comparison with fully-supervised… · p. 11
D. Robotic Experiment: User Study in learning trajectories
- The user study provides a proof of concept that re-ranking and zero-G or interactive feedback can train robots in few feedback iterations.The authors note that the study was limited in size.Source: D. Robotic Experiment: User Study in… · pp. 11, 13
- Five users trained Baxter and two trained PR2 on 10 tasks, but the PR2 study required Rviz-ROS familiarity.The study used zero-G and re-ranking feedback on Baxter, and interactive waypoint correction in place of zero-G on PR2.Source: D. Robotic Experiment: User Study in… · p. 11
- Trajectory quality improved with feedback: on average, a user took 5 feedback to improve over Oracle-svm; per task, users took 3 re-rank and 2 zero-G feedback.As task difficulty increased, zero-G or interactive feedback counts increased, suggesting users relied more on it for difficult tasks to precisely rectify erroneous waypoints.Source: D. Robotic Experiment: User Study in… · p. 11
- Users took an average of 5.5 minutes per grocery task, with training time reported only for that setting.Household feedback required Rviz-ROS experience, and users found simulator waypoint correction difficult.Source: D. Robotic Experiment: User Study in… · pp. 11–12
VII. CONCLUSION AND FUTURE WORK
The paper concludes that coactive feedback from non-experts can teach robots contextual trajectory preferences in household and grocery checkout settings. It identifies uncertainty and broader applications as directions for future work.
VII. CONCLUSION AND FUTURE WORK
- The approach learns preferences beyond geometric constraints through iterative improvements from non-expert users, without requiring optimal demonstration trajectories.The authors evaluated it with PR2 in household settings and Baxter in grocery checkout settings, where experiments suggest training takes a few minutes and a few feedbacks.Source: §VII · p. 13
- Extending coactive learning to uncertain object poses or attributes would require a belief-space update, with different theoretical guarantees.The authors identify this as a possible extension.Source: §VII · p. 13
- The authors also identify assistive cars, crowdsourced feedback, and other easy-to-elicit learning signals as possible future directions.Source: §VII · p. 13
APPENDIX A PROOF FOR AVERAGE REGRET
The appendix bounds TPP’s average regret under strictly α-informative feedback, assuming bounded feature vectors and a scoring function that contains the user’s hidden score function. The proof combines the weight updates with inequalities to derive the bound.
APPENDIX A PROOF FOR AVERAGE REGRET
- The proof assumes the hidden user score function belongs to the scoring-function family and that feedback is strictly α-informative.The feedback condition is later relaxed to hold only in expectation.Source: Appendix A · pp. 13–14
- The theorem bounds TPP’s average regret under the stated feedback assumption, with a constant C bounding the combined feature-vector norm.Source: Appendix A · p. 14
- The proof applies Cauchy–Schwarz and the weight updates to bound terms in the regret expression, then combines the resulting inequalities.It rewrites the feedback condition before combining the intermediate bounds.Source: Appendix A · p. 14
APPENDIX B PROOF FOR EXPECTED REGRET
The appendix extends the regret analysis to expected α-informative feedback. It takes expectations in the proof inequalities and uses Jensen’s inequality to obtain the expected-regret bound.
APPENDIX B PROOF FOR EXPECTED REGRET
- The corollary bounds TPP’s expected regret under expected α-informative feedback.The expectation is over feedback choices when the current task and trajectory are known.Source: Appendix B · p. 14
- The proof takes expectations of the earlier inequalities and applies Jensen’s inequality to the concave square-root function.Source: Appendix B · p. 14
- The proof then uses the expected α-informativeness condition to obtain the stated regret bound.Source: Appendix B · p. 14