PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching Robot Policies to Humans Using Erroneous Examples

Rithika Narayan, Suresh Kumaar Jayaraman, Henny Admoni

arXiv:2608.29023v1cs.RO

TL;DR

Teaching humans nuanced robot policies remains an open problem, despite extensive demonstration-based work and educational inspiration. This paper extends an existing teaching framework with erroneous demonstrations and self-explanations, then evaluates them in a user study. The findings suggest improved policy retention over time and indicate that inverse reinforcement learning-like reasoners perform best on prediction tasks.

  • Problem

    Teaching humans nuanced robot policies remains unresolved, despite prior demonstration-based approaches and educationally inspired interventions.

  • Method

    The paper integrates self-explanations and erroneous examples into an existing robot policy teaching loop and evaluates them in a user study.

  • Results

    Learners who most commonly used inverse reinforcement learning-like reasoning outperformed peers using other reasoning styles on optimal response and regret metrics.

  • Takeaways & Limitations

    Erroneous demonstrations and verbalized reasoning are promising tools for teaching robot policies, while learner reasoning style is relevant to policy prediction performance.

  • Takeaways & Limitations

    Treatment-group differences were not statistically significant, partly attributed to the small sample and high performance variance.

Abstract

from arXiv · show

Human-robot collaboration describes the process of humans and autonomous agents working together to accomplish common goals. This process is facilitated best when robot policies, or behaviors in different situations, are made transparent to humans. Demonstration-based explanations have been a focus of human-robot collaboration research, and the field has frequently drawn upon literature from education to improve how humans are taught robot policies. However, no single teaching method has been proven effective across domains, difficulties, learners, and other variables; the question of how humans can most effectively be taught robot policies remains open. In traditional classrooms, learners are shown erroneous examples, in which they reflect on and correct incorrect responses to understand common pitfalls when learning a concept. We propose using erroneous examples to teach robot policies, extending an existing policy teaching framework. We conduct a user study in which participants view incorrect demonstrations of robot behavior and correct the actions to align with the actual policy. Our findings suggest that viewing these incorrect demonstrations and verbalizing one's reasoning in predicting a robot's actions improves retention of the policy over time, in agreement with the effect of erroneous examples in classrooms. We also categorize participants into distinct learning styles and establish that participants using inverse reinforcement learning-like reasoning perform best on policy prediction tasks. With this work, we aim to advance the methods by which robots educate humans on their policies.

Introduction

Humans need transparent robot policies to collaborate safely and effectively, yet nuanced policies remain difficult to teach. This paper integrates self-explanations and erroneous examples into demonstration-based policy teaching and studies learner characteristics.

  • Motivation: Transparent robot policies support human trust, safety, and effective collaboration with autonomous robots.Understanding robot decision-making can help people verify behavior and anticipate actions in collaborative tasks.
  • Robot Policies: Robot policies are situation-dependent rules, often shaped by tradeoffs among rewards and costs.Examples include avoiding stairs or balancing travel distance against environmental hazards.
  • Prior Work: Demonstrations convey complex reward functions more effectively than mathematical definitions or verbal explanations, but nuanced policies remain difficult to understand.Prior work augments demonstrations with tests and feedback, yet reports a ceiling in human understanding.
  • Contribution: The proposed framework adds self-explanations and erroneous, suboptimal demonstrations to the robot policy teaching loop.Learners verbalize reasoning and inspect incorrect behavior while learning to predict robot trajectories in new environments.
  • Contributions: The paper contributes a user study of these techniques and a characterization of human learners into distinct learning styles.It also validates assumptions used in modeling human beliefs about robot policies.

Related Works

Prior work combines explainable reinforcement learning with educational strategies to make complex robot policies understandable. The paper builds on demonstration-based teaching, scaffolding, remedial correction, and MDP-based policy modeling.

  • Explainable Reinforcement Learning: Explainable reinforcement learning aims to make reward-driven agent policies understandable without overwhelming people with numerical rewards and costs.Prior approaches also use inherently interpretable models for complex policies and environments.
  • Educational Strategies: Educational scaffolding incrementally increases demonstration difficulty so learners can approach complex policies without becoming overwhelmed.This strategy is especially relevant when several environmental features influence behavior.
  • Teaching Framework: The prior demonstration-based teaching loop provides the basis for the proposed framework.The paper adapts classroom-inspired techniques to policy demonstrations and learner correction.
  • Study Setting: The user study models two robots as Markov Decision Processes with shared reward, action, and discount components within a domain.Domains differ in transition, state, and initial-state distributions.

2.3 Machine Teaching for Policies

The machine-teaching framework selects informative demonstrations while modeling learners as recovering reward weights through inverse reinforcement learning and planning. Demonstrations become reward constraints that define taught knowledge components.

  • Machine Teaching: The framework selects demonstrations that help learners recover a robot’s optimal policy while modeling learning through inverse reinforcement learning.Learners approximate optimal reward weights and plan within the MDP to predict behavior.
  • Behavior Equivalence Classes: A Behavior Equivalence Class is the set of policies for which a given demonstration would be optimal.For a demonstration, the BEC is represented as a half-space constraint in reward-weight space.
  • Behavior Equivalence Classes: The IRL equation converts a demonstration into a constraint on the reward vector.The resulting constraint represents a knowledge component, such as a tradeoff between path length and distance from furniture.
  • Educational Foundations: Self-explanation is a classroom-derived technique in which learners verbalize their reasoning while solving a problem.Educational studies associate it with improved reasoning and concept retention, while open-ended responses avoid the cognitive burden reported for some structured formats.

2.6 Learning from Erroneous Examples in Educational Literature

Educational research motivates teaching robot policies with erroneous examples and self-explanation. The paper adapts these practices within a policy-teaching framework and models learner beliefs to choose informative demonstrations and tests.

  • Erroneous Examples: Erroneous examples ask learners to inspect and correct incorrect solutions, improving long-term concept retention across classroom settings.Examples include deliberately reversing sine and cosine when teaching trigonometry.
  • Transfer to Robot Policies: The paper tests whether classroom benefits from erroneous examples transfer to teaching humans robot policies.The proposed framework incorporates erroneous robot behaviors and evaluates learners’ policy understanding.
  • Study Design: The methodology combines policy-teaching adaptations with a user study examining self-explanations and erroneous examples.The framework also uses modeling techniques to approximate the learner’s understanding of the policy.
  • Belief Modeling: A particle filter represents beliefs over possible reward-weight combinations and updates them after demonstrations and tests.Particles are weighted according to consistency with demonstration-imposed constraints, using uniform and von Mises-Fischer components.

3.2 Robot Policy Teaching Loop

The study adapts a partial-feedback robot policy teaching loop by comparing standard prediction tests with self-explanation and erroneous-example formats across two masked domains.

  • Teaching loop: The teaching loop combines demonstrations of optimal trajectories, unit tests in unseen environments, and feedback on incorrect responses.Demonstrations show action-by-action behavior; unit tests ask learners to provide the optimal trajectory and feedback contrasts incorrect and optimal responses.
  • Study domains: Participants learned taxi and skateboard policies modeled as Markov Decision Processes, with domains visually masked to reduce bias from prior knowledge.The taxi task involved delivery, recharge stations, and mud; the skateboard task involved reaching a destination with optional reduced-cost movement.
  • Study design: All groups received the same demonstrations, tests, and unseen environment configurations; only the unit-test prompt and response format differed.The three treatment conditions were compared between subjects, while both domains were presented to every participant in randomized order.
  • Unit-test conditions: The control condition asked learners to predict robot behavior, whereas the self-explanation condition additionally required written reasoning.The control format matched prior work, while self-explanation was included as a distinct condition for learning robot policies.
  • Unit-test conditions: The erroneous-example condition presented an incorrect trajectory and required learners to explain its error, provide the correct trajectory, and explain their reasoning.This format separates error explanation from explanation of the learner’s own corrected response, following educational erroneous-example practices.
  • Assessment: Immediate and delayed post-tests assessed policy retention by asking participants to predict robot actions in unseen environments 24 to 36 hours later.The delayed assessment was added to measure retention, because educational benefits of erroneous examples often emerge over time rather than immediately.

Results

The study collected immediate and delayed post-test data from participants randomly assigned to three testing conditions across two robot-policy domains.

  • Participants: 39 participants completed the first study part, while 30 returned for the delayed post-test 24 to 36 hours later.The first part included demonstrations, unit tests, feedback, and the immediate post-test.
  • Assignment: Treatment conditions and the order of the two domains were randomly assigned.The groups included 14 control participants, 13 self-explanation participants, and 12 erroneous-example participants in part one.

4.1 Performance in Part One and Part Two

Test condition did not significantly affect optimal responses in either post-test, while domain difficulty differed in the immediate assessment and regret was examined as a finer-grained measure.

  • Immediate post-test: F(1, 39) = 0.65, p = 0.528: test condition had no significant effect on optimal responses in the immediate post-test.Participants performed substantially better in the taxi domain than in the skateboard domain, whose reward and cost tradeoffs were tighter.
  • Delayed post-test: F(1, 30) = 0.136, p = 0.873: test condition had no significant effect on optimal responses in the delayed post-test.Domain also had no significant delayed-post-test effect, with F(1, 30) = 1.702 and p = 0.202.
  • Interpretation: The authors attribute limited room for change partly to a ceiling in taxi performance and a floor in skateboard performance during the first post-test.Very high initial taxi performance leaves little room to improve, while poor initial skateboard performance leaves little room to worsen.
  • Measures: Regret was analyzed separately because optimal-response accuracy can be too coarse to capture meaningful differences from the optimal trajectory.Differences between a learner’s trajectory and the optimal response may indicate understanding even when the exact optimal trajectory is not produced.

4.2 Change Between Immediate and Delayed Post-tests

Across the immediate and delayed post-tests, self-explanation and erroneous-example groups showed retention-related improvements, but the corresponding condition effects were not statistically significant.

  • Optimal responses: The self-explanation and erroneous-example groups improved their mean number of optimal responses from immediate to delayed testing, while the control group worsened.This pattern is shown in Figure 4.1.
  • Regret: The self-explanation group had the greatest decrease in regret, while the control group’s cumulative regret increased.Figure 4.2 indicates that self-explanation trajectories deviated less from optimal trajectories over time.
  • Statistical tests: F(1, 30) = 0.539, p = 0.590: test condition had no significant effect on change in optimal responses.The reported group trends therefore did not establish a statistically significant condition effect.
  • Statistical tests: F(1, 30) = 1.533, p = 0.234: test condition had no significant effect on change in summed regret.The authors describe the observed trends as consistent with educational findings on concept retention.

4.3 Improved Understanding

Erroneous examples did not significantly improve participants’ ratings of understanding, contrary to H5.

  • H5 was not supported: test condition had no statistically significant effect on ratings of understanding.The mixed ANOVA reported F(1, 39) = 1.434 and p = .252.

4.4 Cognitive Load

The study found no significant cognitive-load effect from test condition, while explanation responses revealed predominantly inverse-reinforcement-learning-like reasoning associated with stronger policy-prediction performance.

  • Cognitive Load: H6 was not supported: test condition had no statistically significant effect on self-reported cognitive load.The mixed ANOVA reported F(1, 39) = 1.076 and p = 0.352.
  • Explanation Styles: The study categorized explanations into imitation learning, inverse reinforcement learning, procedural, understanding, and other styles.Two coders analyzed 363 explanations and achieved 78% agreement, with Cohen’s kappa reported as 0.63.
  • Explanation Styles: IRL-based explanations were dominant, and most participants were classified as IRL learners focused on environmental rewards and costs.The explanations were collected through required prompts rather than optional survey questions, producing denser reasoning data.
  • Learning Performance: IRL-style learners outperformed most other groups on optimal responses and regret in both immediate and delayed post-tests.One participant classified as “other” exceeded the other groups on some metrics but was often similar to IRL learners.

4.6 Discussion - Comparison with Prior Work

The discussion situates the study within classroom-inspired robot-policy teaching, noting difficulty in direct comparison with prior work and evidence that learners reason about policies through inverse reinforcement learning.

  • Comparison with Prior Work: Prior studies found that scaffolding and feedback on unit tests improved human performance in predicting robot actions.The authors note that those studies used larger participant pools and iterated on teaching interventions.
  • Comparison with Prior Work: Direct comparison with prior studies is difficult because they used larger samples and different intervention-development processes.The study nevertheless observed a significant interaction between domain and final test performance on immediate and delayed post-tests.
  • Learning Models: Participants frequently explained reinforcement-policy behavior in terms of rewards or costs associated with environmental features, supporting IRL-based models of human learning.The discussion gives a taxi-domain explanation as an example of reward-and-cost reasoning.

Conclusion

The study develops classroom-inspired methods for teaching robot policies and reports promising retention trends and IRL-like reasoning, while acknowledging limited statistical evidence and open questions about instructional scope and timing.

  • Limitations: No statistically significant treatment-group differences were found across quantitative and qualitative metrics.The authors attribute this limitation partly to the small sample and high variance in the self-explanation and erroneous-example groups.
  • Future Work: Future work should expand the study and examine different explanation formats, erroneous-example placements, test intervals, and delayed assessments.The authors note that classroom research provides no standard interval for distinguishing retained concepts from immediately understood ones.
  • Future Work: Long-term retention requirements may differ between policies for autonomous medical robots and package-delivery robots.The discussion contrasts remembering a surgeon’s robot policy over the long term with remembering a pedestrian-facing delivery robot’s policy.
  • Future Work: Erroneous examples may affect learners differently depending on their prior knowledge.The authors connect this question to educational findings that high-prior-knowledge students may benefit more from erroneous examples in mathematics.
  • Conclusion: The work reports promising policy-retention trends and identifies IRL-like reasoning processes while extending robot-policy teaching with self-explanation and erroneous examples.The conclusion frames these findings as steps toward greater transparency and trust in human-robot collaboration.
Loading 2608.29023v1…