PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Trajectory Preferences for Manipulators via Iterative Improvement

Ashesh Jain, Brian Wojcik, Thorsten Joachims, Ashutosh Saxena

arXiv:1306.6294v2cs.ROcs.AIcs.HC

TL;DR

Manipulation preferences vary across users, tasks, and environments, making good trajectories hard to specify in advance. The paper learns those preferences online from iterative trajectory improvements rather than optimal demonstrations, and reports generalization to unseen tasks and improvement over baselines.

  • Problem

    Trajectory preferences vary with users, tasks, and environments, while optimal trajectories can be difficult to demonstrate for high-DoF manipulators.

    Source: §1 · p. 1

  • Method

    The robot learns a context-dependent trajectory score online by updating its model from user-provided trajectories that improve on its current proposal.

    Source: §3 · p. 2, §4.4 · p. 5

  • Results

    TPP generalizes to unseen tasks and outperforms baseline algorithms; users improved over Oracle-SVM within 5 feedbacks in the user study.

    Source: §5.2 · p. 7, §5.3 · p. 8, §6 · p. 8

  • Takeaways & Limitations

    The reported batch and robotic experiments support learning useful trajectory preferences from a small number of iterative user interactions.

    Source: §5.3 · p. 8, §6 · p. 8

    The approach assumes complete knowledge of the environment; the paper notes object-attribute labeling as one possible source.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

We consider the problem of learning good trajectories for manipulation tasks. This is challenging because the criterion defining a good trajectory varies with users, tasks and environments. In this paper, we propose a co-active online learning framework for teaching robots the preferences of its users for object manipulation tasks. The key novelty of our approach lies in the type of feedback expected from the user: the human user does not need to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system. We argue that this co-active preference feedback can be more easily elicited from the user than demonstrations of optimal trajectories, which are often challenging and non-intuitive to provide on high degrees of freedom manipulators. Nevertheless, theoretical regret bounds of our algorithm match the asymptotic rates of optimal trajectory algorithms. We demonstrate the generalizability of our algorithm on a variety of grocery checkout tasks, for whom, the preferences were not only influenced by the object being manipulated but also by the surrounding environment.\footnote{For more details and a demonstration video, visit: \url{http://pr.cs.cornell.edu/coactive}}

1 Introduction

Good manipulation trajectories must reflect user preferences that vary by task and environment, making them difficult to specify in advance. The paper proposes learning those preferences from iterative trajectory improvements rather than requiring optimal demonstrations.

1 Introduction

  • Trajectory quality depends on user preferences that vary across users, tasks, and environments, not merely geometric feasibility.Examples include keeping a glass upright and moving heavy objects around fragile ones.Source: §1 · p. 1
  • These preferences are difficult to anticipate or manually encode before knowing where and how a robot will be deployed.The paper also notes that optimal trajectories can be challenging to demonstrate on high-DoF manipulators.Source: §1 · p. 1
  • The proposed co-active approach learns preferences from incremental improvements, and the reported evaluation covers 16 grocery-checkout tasks and unseen-task generalization.The introduction reports both batch and robotic experiments with user feedback.Source: §1 · p. 1, Figure 1 · p. 2

2 Related Work

Prior work learns trajectories from demonstrations or plans around specified preferences, while this paper learns trajectory scores from sub-optimal feedback and contextual features.

2 Related Work

  • Learning from demonstration has been used for varied tasks, but providing a full, spatially and temporally consistent arm configuration is challenging on high-DoF manipulators.The paper distinguishes its feedback, which does not disclose the optimal trajectory.Source: §2 · p. 2
  • Some noisy-demonstration methods assume near- or local-optimality, unlike the biased, potentially far-from-optimal relative preferences considered here.The paper also contrasts its modeling of sub-optimal feedback with prior reward-learning work and Calinon et. al..Source: §2 · p. 2
  • The approach models contextual preferences with features for robot configurations, object-object relations, and temporal behavior, unlike prior work lacking such context.It also differs from methods that plan for explicitly specified constraints or optimize trajectory legibility.Source: §2 · p. 2

3 Learning and Feedback Model

The model treats trajectory quality as a user-valued score conditioned on task context. The robot repeatedly proposes a ranked trajectory, receives an improvement or approval, and updates its score model from that feedback.

3 Learning and Feedback Model

  • The context describes the environment, objects, and other relevant inputs, while an unobserved user scoring function assigns trajectory values for that context.The robot learns an approximate scoring function rather than assuming users can provide cardinal valuations.Source: §3 · pp. 2–3
  • The paper motivates relative feedback with the analogy that ranking search results is easier than identifying the best pages for every query.Source: §3 · p. 2
  • At each interaction, the robot ranks sampled trajectories under its current score model, and the user either accepts the top option or supplies an improved trajectory.That feedback indicates the improved trajectory scores higher under the user's hidden function.Source: §3 · p. 3
  • Regret measures the robot's trajectory against the user's optimal trajectory under the true scoring function, which remains unobserved.The paper uses regret to characterize performance over the robot's learning lifetime.Source: §3 · p. 3
  • Re-ranking lets users select a better trajectory from a displayed ordering, while zero-G lets them physically correct waypoints without revealing an optimal trajectory.Zero-G is useful for bootstrapping, escaping correctly ordered lists of bad options, and correcting minor errors.Source: §3 · p. 3

4 Learning Algorithm

The learning algorithm scores trajectories using features of the robot, manipulated object, and environment, then ranks sampled candidates. TPP learns user preferences from feedback trajectories and has regret guarantees under an expected-informativeness condition.

4 Learning Algorithm

  • The scoring function is linear in trajectory features and decomposes into object-interaction and object-environment terms.The two terms have separate weight vectors and feature vectors.Source: §4 · p. 3

4.1 Features Describing Object-Object Interactions

  • Object-interaction scores connect trajectory waypoints to nearby or underneath objects, using distance projections and vertical ordering weighted by object-property pairs.An edge is included when an object is within a minimum-distance threshold or lies below the manipulated object.Source: §4.1 · pp. 3–4

4.2 Trajectory Features

  • Trajectory features combine segmented arm posture, object orientation and oscillation spectra, and object-environment interactions into a 75-dimensional vector.The algorithm assumes complete knowledge of the environment.Source: §4.2 · pp. 4–5

4.3 Computing Trajectory Rankings

  • To make continuous trajectory maximization tractable, the method samples candidates with RRT and sorts them by learned score.The paper uses sampling approaches for high-degree-of-freedom manipulators.Source: §4.3 · p. 5

4.4 Learning the Scoring Function

  • Expected α-informative feedback yields TPP regret guarantees, with slack ξ_t accounting for violations of the feedback condition.TPP updates weights toward the feedback trajectory's feature vector relative to the current top-ranked trajectory.Source: §4.4 · pp. 5–6

5 Experiments and Results

Experiments evaluate trajectory preferences across grocery-checkout tasks and test iterative learning with users on Baxter. TPP generalizes across object and environment changes, while users improve trajectories with few feedbacks.

5.1 Experimental Setup

  • The evaluation covered 16 grocery-checkout tasks and 1,300 expert-rated trajectories, comparing pre-trained and untrained online-learning settings.Absolute trajectory ratings were used for quantitative evaluation, not supplied to the algorithms.Source: §5.1 · pp. 6–7
  • Tasks span manipulation-, environment-, and human-centric preferences, from orientation-insensitive cereal to fragile upright cartons and safe knife distances from people.The study evaluates trajectory quality after grasping, while the robot moves items for checkout.Source: §5.1 · p. 6, Figure 4 · p. 6

5.2 Results and Discussion

  • Across object-only, environment-only, and joint changes, pre-trained TPP starts with higher nDCG@3 than untrained TPP; as feedback increases, both improve to almost identical performance, and joint changes are harder.The nDCG@3 plots are averaged over tasks for all types of activities.Source: §5.2 · p. 7
  • TPP outperforms baseline algorithms despite never receiving optimal feedback, and surpasses Oracle-SVM in fewer than five feedbacks.Oracle-SVM is not updated on the test set and requires expert labels there.Source: §5.2 · p. 7, Figure 5 · p. 8
  • The manual algorithm averaged nDCG@3 of 0.57, while combined trajectory features performed best and individual features captured distinct preferences.Object-trajectory features capture orientation preferences; arm-configuration and object-environment features capture contorted arm configurations and distances from surrounding surfaces.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 Robotic Experiment: User Study in learning trajectories

  • Users averaged 3 re-rank and 2 zero-G feedbacks to train Baxter, improved over Oracle-SVM within 5 feedbacks, and spent an average of 5.5 minutes per task.Re-rank feedback was popular for easier tasks, while users relied more on zero-G feedback as difficulty increased.Source: §5.3 · p. 8

6 Conclusion

The paper presents co-active learning for robot trajectory selection from iterative improvements rather than optimal demonstrations. Its algorithm has provable regret bounds, trajectory features generalize to unseen tasks, and robotic experiments find incremental feedback feasible with good learning after only a few iterations.

  • Robotic experiments confirmed incremental feedback was feasible and led to good learning results after only a few iterations.Source: §6 · p. 8
  • The TPP learning algorithm has provable regret bounds despite requiring only weak feedback.Source: §6 · p. 8
  • The proposed trajectory features generalize well to tasks the robot has not seen before.Source: §6 · p. 8
Loading 1306.6294v2…