PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Suboptimal Demonstration via Self-Supervised Reward Regression

Letian Chen, Rohan Paleja, Matthew Gombolay

arXiv:2010.11723v3cs.ROcs.LG

TL;DR

IRL methods for learning from demonstrations often assume users provide optimal or stochastically optimal demonstrations, limiting their fit to suboptimal demonstrations. SSRR characterizes the relationship between injected noise and policy performance to learn an idealized reward, with Noisy-AIRL improving robustness to covariate shift. The authors report higher reward correlation and policy improvement than prior work, and a table-tennis policy exceeding user demonstrations on return speed and topspin.

  • Problem

    IRL methods rely on optimality assumptions, while D-REX's Luce-Shepard ranking treats noise levels as ordinal and does not accurately reflect noise-performance degradation.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    SSRR characterizes how policy performance relates to injected noise to learn an idealized reward, and Noisy-AIRL improves robustness to covariate shift induced by noise injection.

    Source: §3 · p. 3, §4.2 · p. 5

  • Results

    Reward correlation was ∼0.95 for SSRR versus ∼0.75 for prior work, and policies achieved ∼200% improvement over suboptimal demonstrations.

    Source: §8 · p. 8

  • Takeaways & Limitations

    In a real-world table-tennis demonstration, the SSRR policy achieved 32% faster return speeds and 40% more topspin than user demonstrations.

    Source: §1 · p. 2

    AIRL's estimated reward may be unreliable for trajectories from a policy different from its current policy, which matters when discriminating trajectories with varying noise and optimality.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Learning from Demonstration (LfD) seeks to democratize robotics by enabling non-roboticist end-users to teach robots to perform a task by providing a human demonstration. However, modern LfD techniques, e.g. inverse reinforcement learning (IRL), assume users provide at least stochastically optimal demonstrations. This assumption fails to hold in most real-world scenarios. Recent attempts to learn from sub-optimal demonstration leverage pairwise rankings and following the Luce-Shepard rule. However, we show these approaches make incorrect assumptions and thus suffer from brittle, degraded performance. We overcome these limitations in developing a novel approach that bootstraps off suboptimal demonstrations to synthesize optimality-parameterized data to train an idealized reward function. We empirically validate we learn an idealized reward function with ~0.95 correlation with ground-truth reward versus ~0.75 for prior work. We can then train policies achieving ~200% improvement over the suboptimal demonstration and ~90% improvement over prior work. We present a physical demonstration of teaching a robot a topspin strike in table tennis that achieves 32% faster returns and 40% more topspin than user demonstration.

1 Introduction

Learning from Demonstration aims to let end-users teach robots through demonstrations, but many IRL methods rely on optimality assumptions that suboptimal demonstrations do not satisfy. SSRR addresses this setting by learning an idealized reward from suboptimal demonstrations.

1 Introduction

  • LfD lets end-users program robots through demonstrations, while IRL seeks the task objective rather than merely imitating demonstrated actions.The introduction contrasts IRL with imitation learning, which learns a state-to-action mapping.Source: §1 · p. 1
  • Maximum Margin IRL assumes optimal demonstrations, while probabilistic IRL relaxes this to stochastic or soft optimality but generally cannot produce a much better policy from suboptimal demonstrations.The passage says probabilistic IRL can recover the optimal reward and policy when demonstrations are only slightly suboptimal under its assumption.Source: §1 · pp. 1–2
  • D-REX uses behavioral cloning and Luce-Shepard trajectory rankings, but the authors identify counterproductive inductive bias and brittleness to covariate shift in its rollout generation.The authors introduce SSRR as a framework addressing these limitations.Source: §1 · p. 2
  • Reward correlations were 0.94, 0.94, and 0.97 for SSRR across HalfCheetah, Hopper, and Ant, versus 0.854, 0.797, and 0.621 for prior work.The comparison uses ground-truth reward correlation.Source: §1 · p. 2
  • Policies trained with SSRR improved over suboptimal demonstrations by 163%, 192%, and 141% across three simulated tasks, versus 49%, 89%, and 39% for prior work.The passage also reports a real-world table-tennis policy with 32% faster returns and 40% more topspin than user demonstrations.Source: §1 · p. 2
  • SSRR addresses suboptimal demonstrations by characterizing how policy performance relates to injected noise and using that relationship to learn an idealized reward.The method also includes Noisy-AIRL to synthesize trajectories and improve reward robustness.Source: §1 · p. 2

2 Preliminaries

The preliminaries define the MDP and IRL setting used by the paper, then describe D-REX as an IRL method that ranks noise-perturbed behavioral-cloning trajectories. The authors identify its noise-ranking assumption as unhelpful and contrast it with learning the noise-performance relationship.

2 Preliminaries

  • An MDP specifies state and action spaces, rewards, transition probabilities, a discount factor, and an initial-state distribution; RL seeks to maximize cumulative discounted reward with an entropy bonus.The maximum-entropy objective favors stochastic policies and encourages exploration during training.Source: §2 · p. 2
  • IRL takes an MDP without a reward and demonstration trajectories as input, then returns a recovered reward; AIRL trains a discriminator against policy rollouts and trains the policy to maximize its pseudo-reward.The AIRL reward function is parameterized by θ.Source: §2 · pp. 2–3

2.1 Disturbance-based Reward Extrapolation

  • D-REX adds uniform noise to a behavioral-cloning policy, assumes higher-noise trajectories rank lower, and learns a reward from pairwise Luce-Shepard rankings.The authors argue this assumption is unhelpful and instead infer the relationship between noisy trajectories and performance.Source: §2.1 · p. 3

3 Characterizing the Noise-Performance Curve for Reward Regression

This section shows that D-REX’s ordinal noise-ranking assumption can misrepresent trajectory rewards, then characterizes noise–performance curves and proposes using a sigmoid-based relation for reward regression.

3.1 D-REX’s Homogeneous Assumption on Noise-Performance Relationship

  • D-REX’s loss treats noise-level intervals as ordinal, yielding r2 = r1+r3 for three trajectories with differing noise levels.The same middle-reward relation results for noise levels 0%, 50%, and 100% as for 0%, 1%, and 100%, although r2 generally depends on η2.Source: §3.1 · p. 3

3.2 The Empirical Noise-Performance Relationship

  • Across five Atari games and five MuJoCo tasks, performance depends on noise interval and environment, and the relationship is well characterized by a four-parameter sigmoid.Figure 1 uses 5 equally spaced noise levels from 0 to 1 for Atari and 20 for MuJoCo; shaded ground-truth regions show standard deviations over 20 trials.Source: §3.2 · p. 4, Figure 1 · p. 4
  • M = 0.996 (SD = 0.004) for the sigmoid method versus M = 0.812 (SD = 0.153) for D-REX on average reward-fit R².The measure is the R² between ground-truth reward and the learned reward function.Source: §3.2 · p. 4
  • The proposed framework synthesizes data to predict an environment- and policy-dependent noise–performance relation, assumes a sigmoid shape, and uses it to infer reward and learn a better-than-demonstration policy.Source: §3.2 · p. 4

4 Our Approach

SSRR uses suboptimal demonstrations to learn idealized rewards and high-performing policies through noise-performance characterization and reward learning. Noisy-AIRL broadens AIRL’s training distribution to improve reward estimates for noisy-policy trajectories.

4.1 Self-Supervised Reward Regression

  • SSRR has three phases: self-supervised data generation, noise-performance characterization, and reward and policy learning.AIRL supplies an initial reward and policy; noisy-policy trajectories are evaluated, then a sigmoid is fit to estimated cumulative rewards as a low-pass filter.Source: §4.1 · p. 5
  • SSRR regresses an idealized reward over trajectory states and actions, then applies reinforcement learning to obtain a policy intended to outperform the suboptimal demonstration.The learned reward is designed to account for noise level and predict policy-performance degradation rather than remain static as in prior work [19].Source: §4.1 · p. 5

4.2 An Improved Self-Supervised Data Generator: Noisy-AIRL

  • Noisy-AIRL injects noise into AIRL’s generator and adjusts the discriminator objective to expose the reward function to a broader state space.The approach targets AIRL reward estimates that may be unreliable for trajectories from policies other than the current policy.Source: §4.2 · pp. 5–6
  • More accurate initial reward estimates from Noisy-AIRL lead to a more accurate noise-performance curve and higher-performing policies.The paper contrasts this IRL-based use of noise with prior noise injection for imitation learning [29].Source: §4.2 · p. 6

5 Results

Across simulated tasks, SSRR more accurately recovers reward functions than D-REX and trains policies that outperform both suboptimal demonstrations and D-REX. The experiments compare reward correlation and policy reward, using multiple synthetic-data generation methods.

5 Results

  • The comparison evaluates ground-truth reward correlation on training and test data, plus ground-truth rewards of policies trained from learned rewards.The methods were tested with synthetic data generated through BC, AIRL, or Noisy-AIRL, with five datasets per method.Source: §5 · p. 6

5.1 Reward Results

  • SSRR + Noisy-AIRL reached reward correlations of 0.941, 0.940, and 0.970 across three tasks, versus 0.845, −0.023, and 0.621 for D-REX.Table 1 compares learned-reward correlations with ground-truth reward across three domains and three synthetic-data generation methods.Source: §5.1 · p. 6, Table 1 · p. 7
  • SSRR’s trajectory-ranking accuracy exceeded D-REX’s on HalfCheetah (87.5% vs. 79.4%), Hopper (85.4% vs. 80.2%), and Ant (88.0% vs. 68.13%).D-REX’s reward correlation varied from 0.621 on Ant to 0.845 on HalfCheetah-v3, while SSRR captured domain-specific noise-performance relationships.Source: §5.1 · p. 7

5.2 Policy Results

  • 199% average improvement over suboptimal demonstrations and 87.5% over D-REX were achieved by policies trained with SSRR rewards.The policy was trained with Soft Actor-Critic, using the highest-correlation reward function from five repeats for each method.Source: §5.2 · p. 7, Table 2 · p. 7

6 Real-world Demonstration: Table-Tennis

The authors taught a Sawyer robot a topspin table-tennis strike using five suboptimal demonstrations, then evaluated return speed, spin, and reward. The SSRR policy outperformed the demonstrations and AIRL trajectories on all three metrics.

6 Real-world Demonstration: Table-Tennis

  • Five kinesthetic demonstrations were suboptimal: their strikes narrowly cleared the net, traveled more slowly, and produced less topspin.The strike was assessed by longitudinal return speed and net downward acceleration as a measure of topspin.Source: §6 · p. 7
  • SSRR policy trajectories had higher return speed, more spin, and higher reward than five suboptimal demonstrations and five AIRL trajectories.The comparison also included ten successful SSRR trajectories; the reported differences were significant for speed and spin (p < .05) and reward (p < .01).Source: §6 · p. 8, Table 3 · p. 8
  • Figure 4 presents a frame-by-frame illustration of the learned Sawyer table-tennis policy.Source: §6 · p. 7, Figure 4 · p. 8

7 Related Work

Related work distinguishes imitation of demonstrated behavior from inference of the demonstrator’s underlying objective, while noting that common optimality assumptions often fail for human demonstrations. Prior approaches use trajectory rankings, confidence scores, or noise-injected policies; earlier table-tennis research developed models, features, and rewards for robot skill learning.

7 Related Work

  • IL learns a state-to-action model and imitates behavior, whereas IRL seeks the task’s underlying objective; optimality assumptions are often unrealistic for human demonstrations.Prior work identifies sources of suboptimality including unnecessary or incorrect actions, limited scenarios, and wrong intentions.Source: §7 · p. 8
  • Suboptimal-demonstration methods use trajectory rankings, expert/non-expert labels, confidence scores, pairwise human comparisons, or noise-injected policies.The cited work notes that rankings can be costly or error-prone, labels lose granularity, confidence scores can be inaccurate, and human pairwise rankings are time-intensive and potentially noisy.Source: §7 · p. 8
  • Earlier robot table-tennis research developed sophisticated models, features, and rewards to train versatile robot skill-learning policies.Source: §7 · p. 8

8 Conclusion

The paper concludes that SSRR uses the relationship between policy performance and injected noise to learn from suboptimal demonstrations, with Noisy-AIRL improving robustness to covariate shift. The appendix describes its Sawyer table-tennis setup and noise-injection implementation.

8 Conclusion

  • SSRR learns from suboptimal demonstrations by characterizing the relationship between policy performance and injected noise; Noisy-AIRL improves robustness to noise-induced covariate shift.The authors report approximately 0.95 reward correlation with ground truth versus approximately 0.75 for prior work, and approximately 200% improvement over the suboptimal demonstration.Source: §8 · p. 8

A.1 Experiment Setup

  • The table-tennis setup used a seven-degree-of-freedom Sawyer arm, two GoPro cameras, and a Robo-Pong ball feeder with empirically set, constant ball angle and speed.Source: §A.1 · p. 12
  • Robot joint positions and velocities were recorded at 100 Hz, while stereo cameras streamed at 60 Hz for ball tracking and state estimation.The vision pipeline used stereo triangulation, OpenCV detection, and an Extended Kalman Filter.Source: §A.1 · p. 12

A.2 Implementation Details and Hyperparameters

  • Table 4 lists AIRL and Noisy-AIRL table-tennis hyperparameters; SSRR hyperparameters matched those used in simulation.Source: Table 4 · p. 12, §A.2 · p. 12
  • For robot speed-mode control, Gaussian noise was injected into joint-speed commands at nine levels; observed performance degraded from level 0 to level 8.Source: §A.2 · p. 12

A.3 Results

In the table-tennis experiment, the SSRR policy outperformed the average demonstration and AIRL on reported strike outcomes. Its lateral speed, vertical acceleration, and reward also differed significantly from both comparison groups.

A.3 Results

  • The comparison used five suboptimal demonstrations, five AIRL trajectories, and ten successful SSRR trajectories, with one-way ANOVA and a Dunn–Šidák-corrected post-hoc test.The authors report checking normality and homoscedasticity before applying these tests.Source: §A.3 · pp. 12–13
  • SSRR differed significantly from both demonstrations and AIRL in return lateral speed, while AIRL and demonstrations did not differ significantly.The one-way ANOVA was significant (F(17, 2) = 6.01, p < .05).Source: §A.3 · p. 13
  • SSRR differed significantly from both demonstrations and AIRL in vertical acceleration, while AIRL and demonstrations did not differ significantly.The paper measures topspin using net downward acceleration; greater acceleration indicates more topspin.Source: §A.3 · pp. 12–13
  • SSRR reward differed significantly from demonstrations and AIRL, and AIRL reward also differed significantly from demonstrations.The one-way ANOVA was significant (F(17, 2) = 9.10, p < .01).Source: §A.3 · p. 13
  • 75% of SSRR strikes succeeded, compared with AIRL policy’s 20%, and the policy outperformed the average demonstration by a wide margin.Figure 5 depicts the returned-ball arc; a sharper drop is correlated with higher spin.Source: §A.3 · p. 13, Figure 5 · p. 13

B Proof of Theorem 1

This block states a theorem about D-REX’s loss and describes supplementary noise-performance experiments and implementation choices. It also records the study’s environment adjustments, dataset comparisons, and use of trajectory snippets.

B Proof of Theorem 1

  • For three trajectories ordered by increasing noise, the theorem states that optimizing D-REX’s loss yields r2 = r1+r3 for their cumulative rewards.The proof frames the optimization as fixing r1 and r3 while minimizing the loss over r2.Source: B Proof of Theorem 1 · p. 14

C Different Policy Noise-Performance Relationship

D.1 Environment Details

  • Hopper-v3 and Ant-v3 disable “terminate when unhealthy” during training and evaluation to avoid a near-linear return-versus-trajectory-length shortcut.The passage explains that this setting could make any positive constant reward appear highly correlated with ground-truth reward.Source: §D.1 · p. 16

D.2 Experiment Details

  • To compare SSRR and D-REX across noisy-data variation, the authors generated five datasets for each of BC, AIRL, and Noisy-AIRL, then tested both methods on the same five datasets.Source: §D.2 · p. 16

D.3 Snippet

  • D-REX snippets are variable-length consecutive trajectory segments; ranking pairs use same-length snippets to increase the amount of data.The passage also says snippet reward-learning targets are adjusted by a length ratio.Source: §D.3 · p. 16

D.4 Implementation Details and Hyperparameters

The implementation details specify simulation policy training, noise settings, SSRR snippet and regularization parameters, and the scope of a hyperparameter table.

D.4 Implementation Details and Hyperparameters

  • SAC policies were trained using stable-baselines, with partially trained checkpoints supplying suboptimal demonstrations and unseen test trajectories across a low-to-high performance spectrum.The checkpoints were taken at 36,000, 90,000, and 650,000 steps for HalfCheetah-v3, Hopper-v3, and Ant-v3, respectively.Source: §D.4 · p. 16
  • AIRL and Noisy-AIRL used ten trajectory noise levels from 0.0 through 0.25, including five zero-noise trajectories.The passage gives the exact levels as [0.0, 0.0, 0.0, 0.0, 0.0, 0.05, 0.10, 0.15, 0.20, 0.25].Source: §D.4 · p. 16
  • SSRR used snippet lengths of 50–500 for trajectories of length 1,000 and an L2 regularization weight of 0.1.Source: §D.4 · p. 16
  • Table 5 is identified as listing AIRL and Noisy-AIRL hyperparameters for simulation domains.Source: Table 5 · p. 17
Loading 2010.11723v3…