PaperlayerView exact revision on arXiv

Source-linked AI summary

SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning

Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee

arXiv:2203.10050v1cs.LGcs.AI

TL;DR

Preference-based RL avoids costly hand-engineered rewards but often requires substantial human feedback. SURF learns from confident pseudo-labels on unlabeled experiences and temporally cropped behaviors; it significantly improves feedback-efficiency across locomotion and robotic manipulation tasks. The authors note that more efficient preference learning could also reduce effort for teaching harmful behaviors.

  • Problem

    Preference-based RL learns rewards from teacher preferences rather than hand-engineered functions, but its costly human labeling can make application difficult.

    Source: §1 · p. 1

  • Method

    SURF combines confidence-filtered pseudo-labels from unlabeled experiences with temporal cropping of behavior segments for reward learning.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    SURF significantly improves feedback-efficiency over PEBBLE across complex locomotion and robotic manipulation tasks; on a complex manipulation task it reaches ∼100% success while the baseline reaches ∼50% with the same few hundred queries.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SURF makes preference-based learning more tractable across complex robotic manipulation and locomotion tasks by improving feedback-efficiency.

    Source: §6 · p. 9

    More feedback-efficient preference learning may also reduce the effort needed to teach harmful behaviors, motivating attention to safe adaptation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the reward with a supervisor's preference between the two agent behaviors. However, preference-based learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent success of these approaches, we present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. To further improve the label-efficiency of reward learning, we introduce a new data augmentation that temporally crops consecutive subsequences from the original behaviors. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the state-of-the-art preference-based method on a variety of locomotion and robotic manipulation tasks.

1 INTRODUCTION

Hand-engineered rewards require substantial effort and can encode undesirable shortcuts, while preference-based RL avoids reward engineering but still depends on costly human feedback. SURF combines confident pseudo-labeling of unlabeled experiences with temporal cropping, and improves feedback-efficiency on locomotion and robotic manipulation tasks.

1 INTRODUCTION

  • Preference-based RL replaces hand-engineered rewards with teacher preferences, but its costly labeling limits application across tasks.Hand-engineered rewards can require more effort as tasks grow complex, permit undesirable shortcuts, or fail to accommodate different user preferences.Source: §1 · p. 1
  • SURF uses confident pseudo-labels from unlabeled experiences and temporal crops of behaviors to improve reward learning’s feedback-efficiency.The unlabeled experiences can come from past buffer data; temporal crops produce shifted or resized behaviors expected to retain the original preferences.Source: §1 · p. 2
  • SURF’s contribution is a combination of semi-supervised learning and temporal-cropping augmentation not previously evaluated for preference-based RL.The paper motivates the approach with prior work on semi-supervised learning and data augmentation in supervised learning.Source: §1 · p. 2
  • SURF significantly improves feedback-efficiency over PEBBLE on complex locomotion and robotic manipulation tasks, including high-dimensional and partially observable inputs.On a complex robotic manipulation task, SURF reaches ∼100% success using a few hundred preference queries, while the baseline reaches ∼50% under the same condition.Source: §1 · p. 2

2 RELATED WORK

Prior preference-based RL work includes policy initialization and informative-query selection, while RL data augmentation has primarily targeted policy learning or generalization. SURF applies unlabeled data and behavior augmentation to reward learning, alongside established semi-supervised learning approaches.

2 RELATED WORK

  • Prior preference-based RL methods have used demonstrations, unsupervised pre-training, and informative-query selection; SURF instead uses unlabeled samples for reward learning and augments behaviors.The cited query-selection approaches aim to improve feedback-efficiency.Source: §2 · p. 2
  • Earlier RL data augmentation improved policy data-efficiency or generalization, whereas SURF introduces an augmentation method specifically for reward learning.The paper identifies temporal cropping as its reward-learning augmentation.Source: §2 · p. 2
  • Semi-supervised learning uses unlabeled samples when labeled data are limited, with methods combining pseudo-labels, consistency regularization, and data augmentation.The related-work discussion names MixMatch, ReMixMatch, and FixMatch as examples.Source: §2 · pp. 2–3

3 PRELIMINARIES

The preliminaries frame RL as maximizing expected return and preference-based reward learning as supervised prediction from pairwise behavior feedback. The reward model uses accumulated segment rewards to predict teacher preferences and is trained on labeled comparisons.

3 PRELIMINARIES

  • In RL, an agent receives states, chooses actions, transitions between states, and aims to maximize expected return.The return is the discounted cumulative reward, with discount factor γ.Source: §3 · p. 3
  • Preference-based RL learns a reward function consistent with a teacher’s preferences between agent behaviors rather than assuming a hand-engineered reward.The framework treats reward learning as supervised learning from pairs of behavior segments and their preference labels.Source: §3 · p. 3
  • A segment is a sequence of observations and actions, and each labeled pair records which segment is preferred or whether they are equally preferable.The labels are y ∈ {0, 1, 0.5}, for preference of segment 1, preference of segment 0, and equal preference, respectively.Source: §3 · p. 3
  • The Bradley–Terry preference predictor models preference probability from accumulated segment rewards, and the reward function is trained on teacher-labeled data.The model assumes preference probability depends exponentially on the accumulated reward over a segment; training minimizes binary cross-entropy.Source: §3 · p. 3

4 SURF

SURF extends preference-based RL reward learning with unlabeled experience and temporal cropping. It combines confidence-based pseudo-labeling with subsequence augmentation in a framework usable alongside existing preference-based methods.

4 SURF

  • SURF combines unlabeled-sample pseudo-labeling with temporal subsequence cropping and is designed to accompany existing preference-based RL methods.Source: §4 · p. 4

4.1 SEMI-SUPERVISED REWARD LEARNING

  • Pseudo-labeling uses unlabeled segment pairs only when predictions exceed a confidence threshold, with training encouraging high-confidence preferences through entropy minimization.The unlabeled pool is not fixed-size because arbitrary experience pairs can be sampled from the buffer.Source: §4.1 · p. 4

4.2 TEMPORAL DATA AUGMENTATION FOR REWARD LEARNING

  • Temporal cropping creates subsequences of labeled behavior pairs and trains the reward model to preserve their teacher preference across shifted or resized versions.The approach is related to consistency regularization, which trains similar predictions on augmented versions of a sample.Source: §4.2 · p. 5

5 EXPERIMENTS

SURF is evaluated on scripted-teacher robotic manipulation and locomotion benchmarks, ablations of its components, and pixel-based control. Across the reported tasks, it improves on PEBBLE with the same preference-query budget, while component and hyperparameter studies examine when those gains occur.

5.1 SETUPS

  • The experiments use a scripted teacher for quantitative evaluation, PEBBLE as SURF’s backbone, and SAC with ground-truth rewards as an upper-bound comparison.The visual-control extension uses DrQ-v2 with 84 × 84 × 3 image observations.Source: §5.1 · pp. 5–6

5.2 BENCHMARK TASKS WITH SCRIPTED TEACHERS

  • With 400 queries on Window Open, SURF matches SAC’s ground-truth-reward performance while PEBBLE remains far behind; PEBBLE needs 2,500 queries for comparable performance, about six times more.SURF also significantly improves PEBBLE on all six considered manipulation tasks and matches SAC on four.Source: Figure 2 · p. 6, §5.2 · p. 7
  • With 100 or 1,000 queries, SURF significantly improves PEBBLE on Walker-walk, Cheetah-run, and Quadruped-walk.The scripted teacher’s preferences reflect ground-truth rewards, enabling quantitative evaluation by true return.Source: §5.1 · p. 5, Figure 3 · p. 7, §5.2 · p. 7

5.3 ABLATION STUDY

  • On Walker-walk with 100 queries, pseudo-labeling improves PEBBLE, temporal cropping further improves supervised and semi-supervised reward learning, and SURF outperforms PEBBLE across query sizes 50, 100, 200, and 400.The authors report that both components are effective and their combination is essential to SURF’s success; the gain is larger at 50 and 100 queries.Source: §5.3 · p. 7, Figure 4 · p. 8
  • On Walker-walk with 100 queries, temporal cropping outperforms random amplitude scaling and Gaussian noise; Gaussian noise degrades PEBBLE’s performance.Random amplitude scaling improves PEBBLE and is described as an orthogonal augmentation that could be combined with SURF.Source: Figure 4 · p. 8, §5.3 · p. 8
  • On Walker-walk with 100 queries, SURF is robust to most tested unlabeled-batch ratios and confidence thresholds, but performs worse at µ = 7 and τ = 0.95; tuning λ offers more improvement.The experiments use λ = 1, which the authors describe as a simple choice that could be further tuned.Source: Figure 5 · p. 8, §5.3 · pp. 8–9

5.4 EXPERIMENTS ON VISUAL CONTROL TASKS

  • With pixel-based inputs, SURF significantly improves PEBBLE and matches DrQ-v2 using ground-truth rewards on Walker-walk with 200 queries.The authors identify combining temporal cropping with existing image augmentations as future work.Source: Figure 6 · p. 9, §5.4 · p. 9

6 DISCUSSION

SURF combines confident pseudo-labeling with temporal cropping and improves feedback-efficiency across complex manipulation and locomotion tasks. The discussion identifies state-based inputs as a boundary and flags risks from malicious preference corruption.

6 DISCUSSION

  • SURF significantly improves feedback-efficiency on complex robotic manipulation and locomotion tasks through pseudo-labeling and temporal cropping.The authors suggest it could make preference-based learning more tractable for diverse and challenging domains.Source: §6 · p. 9
  • Extending the approach to partially observable or high-dimensional inputs such as pixels remains future work.The authors see unlabeled-sample use and augmentation as potentially useful for representation learning in preference-based RL.Source: §6 · p. 9
  • Maliciously corrupted preferences could make feedback-efficient training require less effort to teach harmful behaviors, motivating safe adaptation.The authors frame this as a possible negative impact, not an observed outcome.Source: §6 · p. 9
  • Implementation details are described in Appendix B, and source code is provided in the supplementary material.Source: §6 · p. 9

A PEBBLE ALGORITHM

PEBBLE combines intrinsic-motivation pretraining, off-policy SAC, and experience relabeling to train with a changing reward model.

A PEBBLE ALGORITHM

  • PEBBLE relabels all buffer experiences whenever the reward model is updated to stabilize learning with a non-stationary reward.Source: A PEBBLE ALGORITHM · p. 14
  • PEBBLE pretrains its policy using intrinsic motivation to maximize state entropy and collect diverse experience.Source: A PEBBLE ALGORITHM · p. 14
  • After pretraining, PEBBLE learns the policy with the off-policy reinforcement-learning algorithm SAC.Source: A PEBBLE ALGORITHM · p. 14

B EXPERIMENTAL DETAILS

The experiments use PEBBLE as SURF’s backbone, with shared training hyperparameters and additional specifications for the reward model, sampling, and task implementations.

B EXPERIMENTAL DETAILS

C ADDITIONAL EXPERIMENTAL RESULTS

Component analyses find gains from both semi-supervised learning and temporal cropping, with their combination improving the PEBBLE baseline further. Additional tests find no further gain from naively combining augmentations and report robustness to cropping-range choices.

Ablation study on Meta-world

Applying temporal cropping with other augmentations

Effects of augmentation intensity

Loading 2203.10050v1…