Source-linked AI summary
Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrations
Michael Laskey, Caleb Chuck, Jonathan Lee, Jeffrey Mahler, Sanjay Krishnan, Kevin Jamieson, Anca Dragan, Ken Goldberg
TL;DR
Robot learning from demonstrations can face a mismatch between training and execution states, while robot-centric corrective labeling can be demanding for supervisors. The paper compares human-centric demonstrations with robot-centric sampling across simulations, theory, and a robot singulation study. It finds that robot-centric sampling’s advantage fades with expressive models, and human-centric sampling can perform better or converge when robot-centric sampling does not.
Problem
When a robot visits states unlike its supervisor’s demonstrations, training error may not reflect execution error; the paper asks whether robot-centric sampling’s benefits outweigh its supervisory challenges.
Source: §I · pp. 1–2Method
The paper compares teleoperated human-centric state-control demonstrations with robot-centric sampling that labels states visited during learned-policy execution.
Source: §I · p. 1Results
With highly expressive models, human-centric and robot-centric sampling are essentially at parity; in a 60-trajectory pilot, robot-centric success was 40% versus 60% for human-centric sampling.
Source: §I · p. 2Takeaways & Limitations
The paper concludes that human-centric sampling can perform equally well or better with highly expressive models and can avoid some robot-centric drawbacks.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Motivated by recent advances in Deep Learning for robot control, this paper considers two learning algorithms in terms of how they acquire demonstrations. "Human-Centric" (HC) sampling is the standard supervised learning algorithm, where a human supervisor demonstrates the task by teleoperating the robot to provide trajectories consisting of state-control pairs. "Robot-Centric" (RC) sampling is an increasingly popular alternative used in algorithms such as DAgger, where a human supervisor observes the robot executing a learned policy and provides corrective control labels for each state visited. RC sampling can be challenging for human supervisors and prone to mislabeling. RC sampling can also induce error in policy performance because it repeatedly visits areas of the state space that are harder to learn. Although policies learned with RC sampling can be superior to HC sampling for standard learning models such as linear SVMs, policies learned with HC sampling may be comparable with highly-expressive learning models such as deep learning and hyper-parametric decision trees, which have little model error. We compare HC and RC using a grid world and a physical robot singulation task, where in the latter the input is a binary image of a connected set of objects on a planar worksurface and the policy generates a motion of the gripper to separate one object from the rest. We observe in simulation that for linear SVMs, policies learned with RC outperformed those learned with HC but that with deep models this advantage disappears. We also find that with RC, the corrective control labels provided by humans can be highly inconsistent. We prove there exists a class of examples where in the limit, HC is guaranteed to converge to an optimal policy while RC may fail to converge.
I. INTRODUCTION
The paper compares human demonstrations with corrective labels on robot-visited states, asking whether robot-centric sampling’s benefits outweigh its supervisory and learning challenges. Across simulations, a human pilot study, and a theoretical example, it finds that the advantage of robot-centric sampling diminishes with expressive models, while corrective feedback can be difficult and convergence is not assured.
I. INTRODUCTION
- Learning from Demonstrations is framed as supervised learning from example trajectories, but training error may not predict execution error when the robot visits a different state distribution.This mismatch can arise when the robot cannot exactly reproduce the supervisor’s policy because of modeling errors or insufficient data.Source: §I · p. 1
- Human-centric sampling collects teleoperated state-control demonstrations, whereas robot-centric sampling labels states visited during execution of a learned policy,,.Robot-centric feedback may require supervisors to predict control effects without observing them, and its trajectories can include states requiring complex recovery maneuvers.Source: §I · pp. 1–2
- With highly expressive models, human-centric and robot-centric sampling are essentially at parity, and the performance gap diminishes as expressiveness increases across 100 randomly generated grid worlds.A point-mass control example also shows cases where robot-centric sampling fails to converge while human-centric sampling converges.Source: §I · p. 2
- The paper asks whether robot-centric sampling’s benefits outweigh its practical challenges when the robot uses expressive models that can significantly reduce training error.Robot-centric trajectories may visit states caused by earlier errors, where corrective actions are more complex than those needed during manual demonstrations.Source: §I · pp. 1–2
- For 60 trajectories in a 10-person pilot study, robot-centric policies had 40% success versus 60% for human-centric policies, with a statistically significant average-performance gap.Post-analysis suggested participants had difficulty providing the retroactive feedback required by robot-centric sampling.Source: §I · p. 2
- In a counter-example, human-centric sampling converges to an optimum with probability 1 as demonstrations increase, while robot-centric sampling has non-zero probability of converging suboptimally.The paper distinguishes this result from prior theory, which guarantees robot-centric sampling only in terms of hindsight regret.Source: §I · p. 2
II. RELATED WORK
Related work covers human-centric and robot-centric learning from demonstrations, their theoretical analyses, and approaches for shaping collected data. The paper then formulates the objective and assumptions for learning a policy that matches a supervisor on the task’s state distribution.
II. RELATED WORK
- Ross et al. show DAgger has error linear in horizon T for strongly convex losses, but expected loss on robot-collected data may be high for complex recovery behaviors.The paper analyzes how this effect can prevent robot-centric sampling from converging to the supervisor’s policy.Source: §II · p. 2
- Prior approaches address difficult robot-centric states through sample weighting or guided supervision, while this work retains an unknown, unmodifiable supervisor and potentially high-dimensional image states.He et al. instead propose changing examples to make them easier for the robot to learn.Source: §II · p. 2
- Human-centric demonstrations may use teleoperation, kinesthetic teaching, or waypoint specification; this paper focuses on teleoperation and compares it with robot-centric retroactive feedback,,.Related work also includes human-centric neural-network demonstrations for driving and peg-in-hole insertion, and robot-centric applications such as forest flight.Source: §II · p. 2
III. PROBLEM STATEMENT AND BACKGROUND
- The learning objective is to match the supervisor’s policy on the specified task, under Markovian, stochastic, stationary dynamics.Stationarity means the next-state probability given a state and control does not change over time.Source: §III · p. 2
- The setup assumes access to a state space, controls, a robot or simulator for sampling trajectories, and a supervisor able to label states, possibly noisily.The objective measures control differences with a surrogate loss and minimizes expected loss over states visited by the robot,.Source: §III · pp. 2–3
- Human-centric learning minimizes empirical risk on state-control pairs from supervisor demonstrations, while robot-centric learning aggregates data from states induced by the current policy,.The latter targets the state distribution the final policy will encounter rather than only the supervisor’s demonstration distribution.Source: §III · p. 3
1) Step 1:
Step 1 fits the policy to the current demonstration dataset by minimizing surrogate loss over its state-control examples.
1) Step 1:
- The step computes θ_k by minimizing surrogate loss on the current dataset of demonstrated state-control pairs.Initially, the dataset consists of the initial trajectory demonstrations.Source: 1) Step 1 · p. 3
- The fitted policy parameter is θ_k, obtained from the current dataset D_k.Source: 1) Step 1 · p. 3
- Each example receives equal weight, regardless of its likelihood under the current policy.Source: 1) Step 1 · p. 3
2) Step 2:
Step 2 rolls out the current policy, obtains supervisor labels for visited states, and adds those examples to the demonstration dataset for the next iteration.
2) Step 2:
- DAgger rolls out the current policy to sample likely visited states and requests a supervisor control label for every visited state.The labels may include a zero-mean noise term.Source: 2) Step 2 · p. 3
- The visited states and labeled controls are aggregated into the next demonstration dataset, and the two steps repeat for K iterations or until task performance is sufficient.Source: 2) Step 2 · p. 3
IV. EMPIRICAL ANALYSIS
The empirical analysis compares HC and RC through grid-world experiments, a linear dynamical-system example, and a real-robot singulation study using DAgger as the RC method.
IV. EMPIRICAL ANALYSIS
- The analysis compares HC and RC across simulated grid worlds, a linear dynamical system, and a real robot task.The real-robot study asks 10 participants to teach singulation; DAgger is the example RC method.Source: §IV · p. 3
- Grid-world experiments vary the robot’s policy class across randomly generated environments.The policy class is varied to study how expressiveness relates to the comparison.Source: §IV · p. 3
- A point-mass control example examines a potential limitation of RC in learning complex recovery actions.The analysis concludes with a pilot study of participants teaching a robot to separate an object from clutter.Source: §IV · p. 3
A. Policy Expressiveness
In grid worlds, RC’s advantage over HC diminishes with greater policy expressiveness; with noisy labels, both methods converge similarly but require more data.
A. Policy Expressiveness
- With depth-100 decision trees that contain the expected supervisor policy, RC and HC converge to the supervisor at the same rate.The gap between the methods becomes negligible in the high-expressiveness condition.Source: Figure 2 · p. 4, A. Policy Expressiveness · p. 4
- RC outperforms HC with a Linear SVM, but neither method reaches the optimal supervisor’s performance.The comparison covers 100 randomly generated environments.Source: A. Policy Expressiveness · p. 4
- With noisy supervisor labels, both methods require more data to converge but reach the expected supervisor’s performance at similar rates.The label noise is applied to observed labels with probability 0.3.Source: A. Policy Expressiveness · p. 4
B. Algorithmic Convergence
A two-region point-mass example shows that RC can fail to converge when it trains across distinct dynamics, while HC remains in the supervisor’s region and converges.
B. Algorithmic Convergence
- The example uses a point mass with double-integrator linear dynamics and two regions with different masses.Region 1 has mass 1 and region 2 has mass 4.Source: B. Algorithmic Convergence · p. 4
- The supervisor switches between region-specific infinite-horizon LQG models, while the robot learns a single linear policy by least squares.The target is the origin, and the robot starts at [−15, −10] with zero velocity.Source: B. Algorithmic Convergence · pp. 4–5
- RC fails to converge because it enters region 2 and tries to fit one linear policy to two region-specific supervisors, while HC converges to the supervisor’s policy.The two regions have different dynamics, and the supervisor uses a separate linear model in each.Source: B. Algorithmic Convergence · p. 5, Figure 3 · p. 5
C. Real-World Problem
In a 10-participant singulation study, HC policies succeed more often than RC policies. Participants provide teleoperated demonstrations for HC and retroactive feedback for RC.
C. Real-World Problem
- Singulation requires pushing one object away from its neighbors until at least one object center is 10 cm or more from every other center.The robot observes the cluttered workspace with an overhead camera and uses the image as its state representation.Source: C. Real-World Problem · p. 5
- HC uses 60 Xbox-controller demonstrations, whereas RC uses 20 initial demonstrations followed by two rounds of 20 retroactive-feedback demonstrations.For RC, participants watched half-speed videos of the robot rollout and supplied translation and rotation feedback.Source: C. Real-World Problem · p. 5
- RC obtains statistically significantly higher surrogate loss than HC in both forward and rotation degrees of freedom.The loss is averaged over 10 held-out demonstrations from each participant’s 60 demonstrations.Source: Table I · p. 6
- HC policies succeed on approximately 60% of held-out initial states, compared with approximately 40% for RC.Each policy was evaluated on 30 held-out initial states; the figure reports a 20% improvement for HC at the end.Source: C. Real-World Problem · p. 6, Figure 6 · p. 6
D. Understanding RC Performance in Practice
RC sampling produced a wider range of robot states than HC, including states requiring complex recovery actions, and RC policies had higher average test error. Participants also struggled to match their teleoperated controls with retroactive feedback, which the analysis suggests can lose the supervisor’s intent.
Policy Complexity
- RC trajectories covered varied states, including ones requiring backward motion or rapid direction changes, unlike HC trajectories concentrated on paths needed for singulation.The contrast is shown for trajectories collected from a single initial state.Source: D. Understanding RC Performance in… · p. 6
- With 60 demonstrations, RC policies had significantly higher average test error in translation and rotation, indicating harder generalization to unseen labels.The authors suggest this may be due to the complexity of corrective actions.Source: D. Understanding RC Performance in… · p. 6
Human Supervision
- Five participants had difficulty matching retroactive feedback to teleoperation; the control-angle scatter was dispersed, and example trajectories showed disagreement.The analysis concludes that RC sampling can lose the supervisor’s intent.Source: Figure 7 · p. 6, D. Understanding RC Performance in… · p. 6
- Five participants each teleoperated five demonstrations and then matched their controls using the RC labeling interface.Source: D. Understanding RC Performance in… · p. 6
- Across participants and trajectories, retroactive-control correlations with teleoperation were 0.60 for rotation and 0.22 for translation.A smaller correlation indicates greater difficulty matching teleoperated controls.Source: D. Understanding RC Performance in… · p. 6
V. THEORETICAL ANALYSIS
The theoretical analysis examines when HC and RC converge to the supervisor and considers the accumulation of error for HC. It identifies environments where HC converges to the supervisor but RC does not.
V. THEORETICAL ANALYSIS
- The analysis identifies environments where HC converges to the supervisor but RC does not.Source: §V · p. 6
- The section also presents a new analysis of error accumulation for HC.Source: §V · p. 6
- The theoretical analysis is intended to help explain the paper’s empirical findings on HC and RC learning from demonstrations.Source: §V · p. 6
A. Algorithm Consistency
The analysis contrasts online regret on states visited by a policy with absolute task performance, then gives an example where HC converges to the optimal policy while RC can converge to a suboptimal one. The construction shows how states generated by an initially poor policy can make corrective demonstrations uninformative.
A. Algorithm Consistency
- Low regret can coexist with globally poor task performance because regret compares a policy with the best policy on its observed rollout states.The car-racing example illustrates this when the rollout follows a dirt road and the policy performs well relative to other actions on that road.Source: A. Algorithm Consistency · p. 7
- An initially poor policy can visit states that remain insufficiently informative for RC to reach the optimum, even with corrective feedback.The theorem identifies an environment and policy class in which HC converges to the best policy while RC has a nonzero chance of a suboptimal limit.Source: A. Algorithm Consistency · p. 7
- HC demonstrations are {L,R,R} with probability µ and {L,L,L} with probability 1−µ, so their action majority converges to L for µ<1/2.After RC selects R, its rollouts produce the supervisor sequence {L,R,R}, preserving R as the majority vote.Source: A. Algorithm Consistency · pp. 7–8
- With probability at least ce^-m, RC converges to R≠θ*, whose expected loss exceeds θ*=L, while HC converges to L as demonstrations grow.The example sets µ=1/4; Figure 8 depicts HC following the optimal Orange path and RC the suboptimal Teal path.Source: A. Algorithm Consistency · pp. 7–8, Figure 8 · p. 7
- Reward-aware action selection can address the problem only when the rollout reward provides sufficient information; more expressive policies may require more data than HC.The paper also names increasing model expressiveness as a possible remedy.Source: A. Algorithm Consistency · p. 8
VI. DISCUSSION AND FUTURE WORK
The paper concludes that HC can perform as well as or better than RC with highly expressive models, while avoiding some RC drawbacks. It also notes that training such models presents challenges.
VI. DISCUSSION AND FUTURE WORK
- With highly expressive learning models, HC policies perform equally well or better than RC policies and can avoid some RC drawbacks.Source: §VI · p. 8
- The paper provides theoretical contributions concerning the performance of both HC and RC methods.Source: §VI · p. 8
- Training complex models such as recurrent neural networks or differentiable neural computers may make low training error or sufficient data difficult to achieve.The stated future-work direction is learning robust Human-Centric policies.Source: §VI · p. 8