PaperlayerView exact revision on arXiv

Source-linked AI summary

Showing versus Doing: Teaching by Demonstration

Mark K. Ho, Michael L. Littman, James MacGlashan, Fiery Cushman, Joseph L. Austerweil

arXiv:2610.14004v1cs.LGcs.AIcs.HC

TL;DR

The paper asks how an agent can demonstrate a task to teach its underlying reward structure, rather than simply perform it. It models showing as Bayesian teaching and finds that people’s demonstrations align with its predictions, while standard IRL algorithms also benefit from showing.

  • Problem

    How can an agent demonstrate a task to teach its underlying reward structure, rather than simply perform the task?

    Source: §1 · p. 1

  • Method

    The authors model showing as Bayesian teaching over trajectories and contrast it with standard optimal planning in Markov Decision Processes.

    Source: §1 · p. 2

  • Results

    Across two experiments, human showing behavior matched the model’s predictions, and standard IRL algorithms benefited from showing rather than doing.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Demonstrators modify trajectories to signal reward information, including by visiting varied safe states, staying on ambiguously safe tiles, and looping.

    Source: §4.3.1 · p. 8

    Future research must examine richer state features and whether more savvy observers can leverage a showing agent’s pedagogical intent for better learning.

    Source: §5 · p. 8

Abstract

from arXiv · show

People often learn from others' demonstrations, and inverse reinforcement learning (IRL) techniques have realized this capacity in machines. In contrast, teaching by demonstration has been less well studied computationally. Here, we develop a Bayesian model for teaching by demonstration. Stark differences arise when demonstrators are intentionally teaching (i.e. showing) a task versus simply performing (i.e. doing) a task. In two experiments, we show that human participants modify their teaching behavior consistent with the predictions of our model. Further, we show that even standard IRL algorithms benefit when learning from showing versus doing.

1 Introduction

The paper asks how intentional demonstration differs from simply performing a task, and whether teaching-oriented trajectories can improve reward learning. It models showing as Bayesian teaching and tests its predictions against human behavior and standard IRL.

1 Introduction

  • Unlike efficient task performance, showing may emphasize actions that communicate the underlying reward structure to a learner.The introduction illustrates this contrast with handwashing after handling raw chicken.Source: §1 · p. 1
  • Teaching examples can be chosen to disambiguate a target concept from other candidates, motivating a teaching-oriented account of demonstration.Psychological and computational work on teaching by example informs the paper’s approach.Source: §1 · pp. 1–2
  • Prior work had not successfully modeled how people naturally teach reward functions by demonstration, while IRL typically treated experts as doing rather than showing the task.The authors pose whether showing changes behavior and whether standard IRL can benefit from it.Source: §1 · p. 2
  • The model applies Bayesian teaching by example to reward learning and predicts demonstrations that distinguish goals or signal feature rewards, sometimes through inefficient trajectories.The authors test these predictions with human participants and train a standard IRL algorithm on model and human trajectories.Source: §1 · p. 2

2 A Bayesian Model of Teaching by Demonstration

The model treats a demonstration trajectory as a communicative act: a teacher chooses paths that help an observer infer the intended reward. It builds on MDP planning, IRL, and Bayesian pedagogy, then predicts patterns that the authors report matching human behavior in two experiments.

2 A Bayesian Model of Teaching by Demonstration

  • Pedagogical Inverse Reinforcement Learning treats trajectories as communicative acts that signal the reward function an observer should learn.It combines IRL with Bayesian models of teaching by example.Source: §2 · p. 2

2.1.1 Markov Decision Processes

  • The MDP specifies states, actions, transitions, rewards, and discounting; the trajectory algorithm considers paths bounded by depth and minimum hypothetical-doing probability.A policy maps states to action distributions, and its value is expected cumulative discounted reward.Source: §2.1.1 · pp. 2–3

2.1.2 Inverse Reinforcement Learning (IRL)

  • The doing model assumes experts select actions with a Boltzmann policy, while observers infer reward functions from trajectory likelihoods under a uniform prior.The inverse-temperature parameter controls action randomness.Source: §2.1.2 · p. 3

2.2 Bayesian Pedagogy

  • Bayesian pedagogy models teachers as choosing examples to help learners infer a target; effective examples distinguish it from alternative concepts.The teacher’s softmax parameter α controls the choice, from random selection at α → 0 to maximally informative choices as α → ∞.Source: §2.2 · pp. 3–4

2.3 Pedagogical Inverse Reinforcement Learning

  • The showing model selects trajectories for their capacity to convey reward, requiring nested reasoning about what an observer infers from doing.Its qualitative predictions matched human behavior in both goal-based and feature-based teaching experiments.Source: §2.3 · p. 4

3 Experiment 1: Teaching Goal-based Reward Functions

Experiment 1 tested whether people choose goal-disambiguating paths when showing a goal-based task. Show participants did so more than Do participants, consistent with the model, while a separate swerving pattern was not predicted.

3 Experiment 1: Teaching Goal-based Reward Functions

  • For the & goal, the model favors up-then-right because that route better distinguishes & from #; low trajectory-length costs can also yield looping.The experiment tested whether participants show goal-based reward functions by choosing paths that disambiguate the goal.Source: §3 · p. 4

3.1 Experimental Design

  • Participants first learned the task, then entered Do or Show conditions; Show participants were asked to help a partner identify the reward location.The same sequence of reward locations was used for all participants.Source: §3.1 · p. 4

3.2 Results

  • Show participants chose more goal-disambiguating paths than Do participants, with a condition effect and no condition-by-goal interaction.The condition effect was F(1, 57) = 16.17, p < .001; the interaction was F(1, 57) = 0.98, p = 0.32.Source: §3.2 · p. 5
  • For the # goal, Show participants swerved more than Do participants, although the model predicted no condition difference for that goal.Swerving was defined as making more than one change in direction; participants may have used it to signal moving directly toward the goal.Source: §3.2 · p. 5

3.3 Discussion

  • Reaching the goal already reveals its location, yet participants still chose paths that better disambiguated their intended goal.The discussion treats this pattern as evidence that people distinguish showing from doing, consistent with the computational framework.Source: §3.3 · p. 5

4 Experiment 2: Teaching Feature-based Reward Functions

Experiment 2 tested showing and doing in a feature-based reward-learning task. Showing produced demonstrations that better conveyed tile values, fit human behavior, and improved learning by standard MLIRL from both human and model trajectories.

4 Experiment 2: Teaching Feature-based Reward Functions

  • The task focused on feature-based reward functions, which let an agent generalize what it learned in one situation to a new one.The three colored tile types could each be safe or dangerous, and their reward assignments were independent.Source: §4 · pp. 5–6

4.3 Teaching Maximum-Likelihood IRL

  • MLIRL learned the true reward function better from Show than Do trajectories, for both human- and model-produced demonstrations.Figure 2 presents learned reward functions and mean distance between learned and true reward-function weights.Source: Figure 2 · p. 6, §4.3 · p. 8

4.1 Experimental Design

  • Participants learned tile values with feedback, then completed no-feedback Do or Show demonstrations; Show participants were told another person would learn from watching.They completed eight learning-and-applying pairs covering the possible safe/dangerous assignments.Source: §4.1 · p. 7

4.2 Results

  • Show participants took paths that signaled tile reward values, while Do participants took efficient routes.Show paths went through multiple safe tile types, remained on safe colored tiles when safe white tiles were available, and looped at boundaries between differently colored safe tiles.Source: §4.2 · p. 7

4.2.1 Model-based Analysis

  • The standard planning model fit Do behavior better, while the pedagogical model fit Show behavior better; the latter was not simply random behavior.The comparison used maximum-median-likelihood parameter values for each model, reward function, and condition combination.Source: §4.2.1 · p. 7

4.2.2 Behavioral Analyses

  • Show participants visited a wider variety of colored tiles, spent a greater share of visits on them, and revisited states more, with no total-reward difference.The analyses treat colored-tile visit entropy as variety and state revisits as an indicator of looping.Source: §4.2.2 · p. 8

4.3.1 Discussion

  • The authors conclude that pedagogical reasoning captures how people show feature-based tasks and that such demonstrations also benefit standard IRL.The described signaling behaviors include visiting varied safe states, staying on ambiguously safe tiles, and looping.Source: §4.3.1 · p. 8

5 General Discussion

The paper presents Bayesian teaching by demonstration and reports that human behavior in two experiments follows its predictions. It also reports benefits for standard IRL algorithms and identifies richer state features and more capable observers as future research directions.

5 General Discussion

  • The Bayesian model of showing makes accurate quantitative and qualitative predictions about human teaching behavior across two experiments.Experiment 1 examined signaling goal information, and Experiment 2 examined teaching feature-based reward functions.Source: §5 · p. 8
  • Standard IRL algorithms benefit from learning from showing rather than merely doing.Source: §5 · p. 8
  • Future work should study showing with richer state features and observers better able to leverage a demonstrator’s pedagogical intent.Source: §5 · p. 8

Appendix 1: An Example with 2 Goals

The example considers a gridworld with two possible goals and a single start. Its analysis restricts attention to length-2 trajectories that end at a goal.

Appendix 1: An Example with 2 Goals

  • The analysis assumes no step costs, sets γ = .99, and considers only length-2 trajectories terminating at a goal.This restriction leaves four trajectories.Source: Appendix 1: An Example with 2 Goals · p. 10
  • The gridworld has two possible goal states, X and Y, and a single starting state.Source: Figure 4 · p. 10
  • Figure 4(ii) shows all length-2 trajectories that terminate at a goal state.Source: Figure 4 · p. 10

Proof

The proof contrasts standard planning with showing in a two-goal example. It establishes that showing favors a trajectory that better disambiguates the intended goal, even when the trajectories reach a goal.

Proof

  • Under standard planning, observing xout provides better evidence for goal X than observing xin.The observer infers the goal using Bayes’ rule, and the proof considers softmax or ϵ-greedy planning distributions.Source: Proof · p. 10
  • PShowing(xout | X) > PShowing(xin | X) in the proof’s example, although both trajectories reach a goal.Source: Proof · pp. 10–11
  • Different probabilities for xout and xin when Y is the goal let a showing agent break their symmetry when X is the goal.Source: Proof · p. 11

Appendix 2: Experiment 2 Model Fits

The appendix identifies a table of Experiment 2 maximum median likelihood model parameters and explains the reward-function code notation.

Appendix 2: Experiment 2 Model Fits

Loading 2610.14004v1…