PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Hypothesis Space Misspecification in Learning from Human-Robot Demonstrations and Physical Corrections

Andreea Bobu, Andrea Bajcsy, Jaime F. Fisac, Sampada Deglurkar, Anca D. Dragan

arXiv:2002.00941v2cs.ROcs.AIcs.HCcs.LGstat.ML

TL;DR

Robots that learn objectives from human input may lack the objective a person intends, and ordinary uncertainty over candidate objectives does not address whether any candidate is correct. This paper infers how well the robot’s hypothesis space explains demonstrations and physical corrections, and reports that situational confidence helps a robot recognize when it cannot explain the input.

  • Problem

    A robot’s hypothesis space may not include the human’s objective, and even uncertainty-aware methods assess candidate hypotheses without checking whether any is correct.

    Source: §VIII · p. 18

  • Method

    The framework jointly reasons about objectives and situational confidence, with demonstrations and corrections as input types.

    Source: §VIII · p. 18

  • Results

    Experiments and a user study with a 7-DoF manipulator show that situational confidence assists the robot in recognizing when it cannot explain human input.

    Source: §VIII · p. 18

  • Takeaways & Limitations

    When human input is poorly explained across objectives, the robot can use its prior rather than changing its objective based on that input.

    Source: 1) Context-dependent usage of… · p. 6

    In expressive hypothesis spaces, misspecification can remain undetected when individual inputs are explained; a single demonstration or data point is insufficient.

    Source: §VIII · p. 18

Abstract

from arXiv · show

Human input has enabled autonomous systems to improve their capabilities and achieve complex behaviors that are otherwise challenging to generate automatically. Recent work focuses on how robots can use such input - like demonstrations or corrections - to learn intended objectives. These techniques assume that the human's desired objective already exists within the robot's hypothesis space. In reality, this assumption is often inaccurate: there will always be situations where the person might care about aspects of the task that the robot does not know about. Without this knowledge, the robot cannot infer the correct objective. Hence, when the robot's hypothesis space is misspecified, even methods that keep track of uncertainty over the objective fail because they reason about which hypothesis might be correct, and not whether any of the hypotheses are correct. In this paper, we posit that the robot should reason explicitly about how well it can explain human inputs given its hypothesis space and use that situational confidence to inform how it should incorporate human input. We demonstrate our method on a 7 degree-of-freedom robot manipulator in learning from two important types of human input: demonstrations of manipulation tasks, and physical corrections during the robot's task execution.

I. INTRODUCTION

Robots learn objectives from human input, but designers cannot anticipate every human preference, so the desired objective may fall outside the robot’s hypothesis space. This paper proposes reasoning about how well that space explains input and demonstrates the approach for both demonstrations and physical corrections.

I. INTRODUCTION

  • Designers’ objective representations can omit human preferences, allowing the robot to misinterpret guidance and behave undesirably.The paper proposes checking whether any hypothesis explains the input, rather than only selecting among available objectives.Source: §I · pp. 2–3
  • The framework quantifies objective-space misspecification and is demonstrated with user demonstrations and an online, close-to-real-time inference algorithm for physical corrections.The correction algorithm is tested in a user study.Source: §I · p. 3

A. Robots learning from humans

  • Prior work learns objectives from demonstrations and corrections, but real human input can be noisy or suboptimal, and uncertainty over objectives does not establish that any objective is likely.These limitations motivate maintaining uncertainty while also assessing how well the hypothesis space explains the input.Source: A. Robots learning from humans · pp. 3–4

B. Uncertainty in robot learning

  • Distribution-based learning methods still assume the desired objective is represented, so they assess which hypothesis is correct rather than whether any hypothesis is correct.The paper focuses on the latter question.Source: B. Uncertainty in robot learning · p. 4

C. Situational confidence estimation

The paper frames objective-space misspecification as the possibility that no parameterized robot cost explains the human’s input. It extends model-confidence reasoning to robot objectives and human inputs on the same dynamical system.

C. Situational confidence estimation

  • This work generalizes model-confidence estimation to robot objectives and studies demonstrations and physical corrections on the same dynamical system.The cited prior work concerns pedestrian motion on separate human and robot systems.Source: C. Situational confidence estimation · p. 4

III. PROBLEM FORMULATION AND APPROACH

  • The robot approximates human preferences with a parameterized cost, trading expressiveness for tractable inference and potentially failing to explain the human’s behavior.The human and robot are assumed to act on the same dynamical system, with the human typically knowing the robot’s control input.Source: §III · pp. 4–5
  • Objective-space misspecification occurs when the human’s cost lies outside the robot’s modeled costs, making it important to assess whether any parameter explains the observed input.The goal is to assess explanation, not merely identify the most likely available hypothesis.Source: §III · p. 5

A. Situational confidence estimation

The framework treats inverse temperature β as an inferred measure of how well the robot’s objective model explains human input. It describes ways to use confidence in interaction and represents costs through basis functions.

A. Situational confidence estimation

  • Inferring β alongside the objective parameter θ makes situational confidence an efficient indicator of how well the robot’s model explains human actions.The approach maintains a joint Bayesian belief and extends Bayesian inverse reinforcement learning with explicit inference over β.Source: A. Situational confidence estimation · p. 5

1) Context-dependent usage of situational confidence:

  • Low confidence can prompt clarification, lead a robot to dismiss an unexplained correction, or inform planning under uncertainty.Full POMDP inference is computationally intractable for large real-world problems; marginalizing over β offers a less demanding planning approach, with poorly explained input potentially leaving the robot to use its prior.Source: 1) Context-dependent usage of… · pp. 5–6

B. Cost representation through basis functions

  • Finite families of basis functions approximate cost functions, but any model can omit components of human preferences, making misspecification relevant beyond hand-crafted features.The paper treats this approach as complementary to open-world objective modeling; comparisons of β values also require unit-norm θ because the observation model is overparameterized.Source: B. Cost representation through basis… · p. 6

IV. ALGORITHMIC APPROACH: DEMONSTRATIONS

The demonstration formulation models human preferences over state trajectories and infers objective parameters alongside situational confidence. Offline inference is made tractable through trajectory sampling and discretization.

A. Formulation

  • Demonstrations are modeled through preferences over state trajectories, and Bayesian inference updates a joint belief over objective parameters θ and situational confidence β.The human controls the demonstrated trajectory; the cost does not directly depend on robot controls.Source: A. Formulation · p. 7

B. Approximation

  • Continuous θ and β with large state and action spaces make exact inference doubly intractable, so the paper uses simple approximations rather than claiming them as contributions.The authors note that other approximate-inference methods could also be used.Source: B. Approximation · p. 7
  • The method samples 1,500 trajectories from random unit-norm θ costs and optimizes them with TrajOpt to approximate the intractable integral.The resulting dynamically feasible trajectories are a practical heuristic for diverse, interesting samples, not a theoretically justified strategy.Source: B. Approximation · p. 7
  • Discretizing θ and β yields a finite posterior that is updated from demonstrations using the sampled trajectories; the method uses a uniform prior but supports any prior.The algorithm takes discretized ΘD and BD, trajectories X, and demonstrations D as input.Source: B. Approximation · p. 7

C. Examples

The examples show that posterior confidence tracks how well the robot’s objective space explains demonstrations. The physical-correction formulation models interventions during execution and requires inference that can run in real time.

C. Examples

  • A perfect, well-explained demonstration concentrates belief on the correct θ and high β; a noisy demonstration of the same objective retains the correct θ but has lower β and a reduced peak.The noisy case still shares the robot’s hypothesis space.Source: C. Examples · p. 8
  • When the robot lacks the cup-away-from-the-human feature, the posterior spreads across θ values and favors low β, indicating poor explanation and uncertainty about the objective.The example’s demonstration is optimal for an objective that includes a feature outside the robot’s hypothesis space.Source: C. Examples · p. 8

A. Formulation

  • During execution, the robot can use human-applied joint torques to update θ and replan its trajectory until another correction arrives or the task ends.The robot initially plans from its current θ and tracks that trajectory with an impedance controller.Source: A. Formulation · p. 9
  • The correction model assumes people care about state-trajectory features, not robot control effort, while trading off a desirable deformed trajectory against their own input effort.The deformed trajectory estimates the human’s desired trajectory from the applied torque.Source: A. Formulation · p. 9
  • Joint inference over continuous θ and β is generally too slow for real-time corrections, motivating an online method beyond the demonstration setting’s simple discretization.The passage notes that discretizing even a small continuous feature space would generally be overly slow for collaborative scenarios.Source: A. Formulation · pp. 9–10

B. Approximation

The approximation makes inference tractable, while the correction method uses estimated situational confidence to moderate objective updates when human inputs fit the robot’s modeled features poorly.

1) Estimating

  • The method estimates β̂ from how efficiently a human correction induces its achieved features: efficient inputs suggest good fit, while inefficient inputs suggest poor fit.The estimate is high when the human’s input is close to the minimal-norm action that achieves those features.Source: 1) Estimating · p. 10

B. Approximation

  • To make inference tractable, the method approximates the constrained input integral with a penalized integral over possible inputs and applies a Laplace approximation.A structural assumption also approximately decouples inference over β and θ.Source: 1) Estimating · pp. 10–11, B. Approximation · p. 10

1) Estimating

  • When E = 1, induced features tend toward low cost under θ; when E = 0, they are modeled as Gaussian noise around the robot’s planned trajectory.The binary variable E indicates whether the intervention is well explained by the modeled cost features.Source: 1) Estimating · p. 10
  • A learned likelihood P(β̂ | E) maps confidence estimates to whether an intervention is explainable, using chi-squared fits to labeled controlled-interaction samples.At deployment, the robot combines this likelihood with a prior to infer P(E | β̂).Source: 1) Estimating · pp. 10–11
  • The confidence-weighted update generalizes common objective-learning formulations, which arise as the special case P(E = 0 | β̂) = 0.After updating θ, the robot replans its trajectory with TrajOpt.Source: 1) Estimating · p. 11
  • Objective updates are weighted by the probability that a correction is well explained, ranging from a full update for certain fit to no update for certain poor fit.This generalizes conventional objective-learning updates, which assume every input is well explained.Source: 1) Estimating · pp. 11–12
  • Online, the robot deforms its trajectory after sensing a human torque, estimates β̂ from the achieved features, updates θ, and replans.The procedure is intended for real-time inference from physical corrections.Source: 1) Estimating · p. 12

C. Examples

Examples contrast fixed-β and estimated-β learning on well-explained and poorly explained corrections, showing how confidence estimation limits unintended learning without recovering an unknown desired behavior.

C. Examples

  • With well-explained inputs, the estimated-β method raises β̂ and learns toward the true θ = 1, matching the fixed-β method’s behavior.The examples contrast corrections toward the table with poorly explained corrections intended to keep a mug upright.Source: C. Examples · p. 13
  • For poorly explained corrections, estimated β̂ reduces unintended learning, while fixed β produces incorrect oscillatory behavior.For well-explained corrections, both methods learn the correct weight θ = 1.0.Source: C. Examples · p. 13
  • Situational confidence estimation reduces unintended learning and undesired behavior, but does not magically recover the user’s correct behavior.Source: C. Examples · p. 13

VI. CASE STUDIES

The case studies apply situational-confidence inference to learning from real human input on a 7-DoF robot manipulator.

VI. CASE STUDIES

  • The case studies use real human input with a 7-DoF robot manipulator.Source: §VI · p. 13
  • They cover learning from demonstrations of manipulation tasks.Source: §VI · p. 13
  • They also cover learning from physical corrections during robot task execution.Source: §VI · p. 13

A. Demonstrations

The demonstrations case study uses human-provided motion-planning examples to test whether inference distinguishes inputs explained by the robot’s feature space from those it cannot explain.

A. Demonstrations

  • Twelve people interacted physically with a JACO 7-DoF arm, and the study analyzed four common personal-robotics cases.Source: A. Demonstrations · p. 13
  • Participants demonstrated moving a cup of coffee from a cupboard to above a table, across a laptop, with respect to features that could be modeled or unmodeled.Features included distance from the table, the person, or a laptop.Source: A. Demonstrations · p. 13
  • Across four scenarios, the robot always modeled three features, while demonstrations could reflect different, potentially unmodeled features.Source: A. Demonstrations · p. 13
  • The study tested whether well-explained input yields high probability on the correct θ and high β, while poorly explained input yields diffuse θ belief and low β.Source: A. Demonstrations · p. 13
  • Researchers first inferred b_single(θ, β) for each demonstration, then combined all 12 demonstrations to estimate b_all(θ, β) for each scenario.Multiple demonstrations can improve confidence in θ estimation.Source: A. Demonstrations · p. 13

1) Well-specified objective space:

Noisy demonstrations can still support the correct objective at relatively high situational confidence, while demonstrations outside the robot’s hypothesis space yield low confidence across its hypotheses.

1) Well-specified objective space:

  • The well-specified case involved demonstrations intended to keep the robot away from the laptop, a feature included in the robot’s hypothesis space.The contrasting misspecified case concerned demonstrations avoiding the human’s body, which the robot’s model did not represent.Source: 1) Well-specified objective space · p. 13, Figure 8 · p. 14, 2) Misspecified hypothesis space · p. 14
  • With all 12 demonstrations, the robot recovered the correct θ with relatively high β despite two noisy demonstrations that came too close to the laptop.Aggregate situational confidence was lower than for the single optimal demonstration: around 1.0 rather than 100.0.Source: 1) Well-specified objective space · p. 14

2) Misspecified hypothesis space:

3) Feature correlation:

Feature correlation can make a missing human-intended objective appear explainable, while difficult demonstrations may require multiple examples to recover the intended feature. In corrections, well-explained actions had higher β, but poorly explained actions still produced feature updates.

3) Feature correlation:

  • Correlated demonstrations favored laptop-distance θ despite the missing human-distance feature, while uncorrelated demonstrations yielded low confidence; this is a limitation.The uncorrelated orange demonstrations produced a uniform distribution over θ hypotheses focused on the lowest confidence values.Source: 3) Feature correlation · p. 15
  • Combining all demonstrations increased posterior probability on laptop-distance θ, but ambiguity in the orange cluster kept confidence below the well-explained case.The blue cluster correlated human-distance avoidance with laptop distance; the orange cluster did not.Source: 3) Feature correlation · p. 15

4) Feature engineering:

  • The table-distance feature’s summed waypoint cost has a counterintuitive optimum: move the hand to the table immediately and keep it above the tabletop.Users found this feature difficult to demonstrate well.Source: 4) Feature engineering · p. 15
  • Users’ gradual approach to the table looked noisy and suboptimal, making the intended θ hard to infer from one demonstration; multiple demonstrations recovered it with reasonably high β.The single-demonstration posterior was less clearly defined around the table-distance feature.Source: 4) Feature engineering · p. 15

B. Corrections

  • F(1, 521) = 9.9093, p = 0.0017: well-explained corrections had significantly higher β, while poorly explained actions still changed features and risked unintended fixed-β learning.The feature differences were non-negligible even for poorly explained interactions.Source: B. Corrections · p. 16, Figure 11 · p. 16

VII. USER STUDY ON LEARNING FROM CORRECTIONS

The user study compared fixed-β and estimated-β learning across well- and poorly-explained corrections, measuring objective learning behavior and participant ratings across four household motion-planning tasks.

VII. USER STUDY ON LEARNING FROM CORRECTIONS

  • The IRB-approved study used 7-dimensional robot trajectories and held object and human locations consistent across tasks and users.The study assumed full observability and accounted for relevant constraints including joint limits and collisions.Source: §VII · pp. 16–17

1) Independent variables:

  • The 2-by-2 design varied learning strategy—fixed-β or estimated-β—and whether corrections were well or poorly explained by the robot’s hypothesis space.Estimated-β learning updated feature weights using the inferred probability of a well-explained interaction.Source: 1) Independent variables · p. 17

2) Dependent measures - objective:

  • Objective measures were regret in feature space and the length of the estimated θ path through weight space; shorter paths and lower regret indicate better learning.Fig. 12 compares regret and path length over time, with lower values preferable.Source: 2) Dependent measures - objective · p. 17, Figure 12 · p. 17

3) Dependent measures - subjective:

4) Hypotheses:

  • The four hypotheses predicted comparable performance or perceptions for well-explained corrections and reduced unintended learning for poorly explained corrections.The latter prediction covered both objective learning and participant perceptions.Source: 4) Hypotheses · p. 17

5) Tasks:

  • Four cup-transfer tasks varied excessive table clearance or grasp angle, with the robot knowing the target feature in Tasks 1 and 3 but not Tasks 2 and 4.Participants physically intervened to correct the robot’s initially incorrect objective.Source: 5) Tasks · p. 17

6) Participants:

In a 12-person within-subjects study, the estimated-β method reduced regret and weight-path length on poorly explained corrections, with no significant difference on well-explained corrections. Participants preferred it for reducing unintended learning, while the discussion identifies important scope and model-detection limits.

6) Participants:

  • The study recruited 12 campus-community participants, aged 18–30; 10 had technical backgrounds and none had used the robot before.The sample comprised six women and six men.Source: 6) Participants · p. 17

7) Procedure:

  • Participants completed both methods in randomized order, correcting an initially incorrect trajectory after seeing the desired trajectory, with two recorded rounds per method.The procedure fixed each task’s ground-truth objective to focus evaluation on objective inference rather than estimating each participant’s unique objective.Source: 7) Procedure · p. 17

1) Objective:

  • For poorly explained corrections, estimated-β significantly reduced regret and average weight paths; on well-explained corrections, neither metric differed significantly.The reported comparisons correspond to Fig. 12a for regret and Fig. 12b for weight-path length.Source: 1) Objective · p. 18

2) Subjective:

  • Participants significantly preferred estimated-β for reducing unintended learning, while task completion and task understanding did not significantly differ from baseline.The survey analysis reports F(1, 7) = 9.15, p = 0.0046 for unintended learning; task completion and understanding had p = 0.348 and p = 0.46, respectively.Source: 2) Subjective · p. 18

VIII. DISCUSSION

  • The discussion frames situational confidence as a measure of how much the robot should trust human input, and reports demonstrations and corrections on a 7-DoF manipulator.The paper presents a general framework for estimating confidence alongside task objectives, including a close-to-real-time approximate algorithm for corrections.Source: §VIII · p. 18
  • In expressive hypothesis spaces, one demonstration or data point may not expose misspecification because a wrong hypothesis can still explain the input; rich, diverse data reduces this chance.The authors identify this as a fundamental detection limitation, especially for more expressive spaces.Source: §VIII · p. 18
  • The experiments cover only simple, few-feature motion planning, and the approach cannot distinguish hypothesis-space misspecification from a misspecified Boltzmann observation model.The paper also does not show how performance degrades under ideal or approximate inference.Source: §VIII · pp. 18–19

APPENDIX A PRACTICAL CONSIDERATIONS

The appendix specifies practical discretizations and planning choices, discusses calibration sensitivities and trajectory deformation, and derives a Laplace approximation for the observation model.

A. Demonstrations

  • For demonstrations, Θ uses 19 normalized vectors from three feature weights in {0, 0.5, 1}, while β uses nine log-spaced values from 0.01 to 100.The threshold ϵ is suggested to be calibrated using a few simulated trajectories.Source: 1) Discretizing Θ and B in (13) · p. 19

1) Planning and Replanning:

  • TrajOpt plans fixed-duration paths between specified start and goal locations while avoiding obstacles; human pushes deform the path and update θ for replanning.Longer paths therefore move faster than shorter ones, and the solver for (21) can return a reasonable estimate from an infeasible initial guess such as 0.Source: 1) Planning and Replanning · p. 19, 2) Solving (21) · p. 19

3) Sensitivity Analysis:

  • λ’s effect is relative when the same value is used for calibration and learning, while ν must avoid extremes because too little precision makes the method always learn without detecting misspecification.Without offline calibration, λ and ν can strongly affect θ and β estimates; automatic calibration is not yet available.Source: 3) Sensitivity Analysis · pp. 19–20

4) Trajectory Deformation Parameter Choice:

  • Trajectory deformation depends on design choices for matrix A and magnitude µ, which may be specified, optimized, or learned from human data.The deformation magnitude can also be tuned for robustness to the rate of deformation.Source: 4) Trajectory Deformation Parameter… · p. 20

APPENDIX B LAPLACE APPROXIMATION IN EQUATION (19)

  • Appendix B approximates the cost to second order around the constrained optimal human action u* to derive a closed-form Gaussian approximation of the observation model.Source: Appendix B · p. 20
Loading 2002.00941v2…