PaperlayerView exact revision on arXiv

Source-linked AI summary

PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation

Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, Xiu Li

arXiv:2306.03615v2cs.LG

TL;DR

Preference-based RL can require costly human labels for each new task, while labels from one task cannot be reused for another. PEARL uses optimal transport to transfer source-task preferences to unlabeled target tasks and models reward uncertainty. Experiments on Meta-World and Robomimic report accurate transfer and significantly better performance than current methods when human preferences are scarce.

  • Problem

    Preference-based RL depends on costly human preference labels, and labels collected for one task cannot be reused for new tasks.

    Source: §1 · p. 1

  • Method

    PEARL aligns source and target trajectories with Gromov-Wasserstein distance, transfers preferences through the transport matrix, and models rewards as distributions to capture uncertainty.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Experiments on Meta-World and Robomimic report accurate zero-shot preference transfer and significantly better performance than current methods when preference labels are scarce.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    PEARL is reported to reduce human labeling costs substantially in its robotic manipulation setting.

    Source: §6 · p. 9

    PEARL may process high-dimensional inputs more slowly and relies on source and target tasks sharing the same action space.

    Source: §6 · p. 9

Abstract

from arXiv · show

In preference-based Reinforcement Learning (RL), obtaining a large number of preference labels are both time-consuming and costly. Furthermore, the queried human preferences cannot be utilized for the new tasks. In this paper, we propose Zero-shot Cross-task Preference Alignment and Robust Reward Learning (PEARL), which learns policies from cross-task preference transfer without any human labels of the target task. Our contributions include two novel components that facilitate the transfer and learning process. The first is Cross-task Preference Alignment (CPA), which transfers the preferences between tasks via optimal transport. The key idea of CPA is to use Gromov-Wasserstein distance to align the trajectories between tasks, and the solved optimal transport matrix serves as the correspondence between trajectories. The target task preferences are computed as the weighted sum of source task preference labels with the correspondence as weights. Moreover, to ensure robust learning from these transferred labels, we introduce Robust Reward Learning (RRL), which considers both reward mean and uncertainty by modeling rewards as Gaussian distributions. Empirical results on robotic manipulation tasks from Meta-World and Robomimic demonstrate that our method is capable of transferring preference labels across tasks accurately and then learns well-behaved policies. Notably, our approach significantly exceeds existing methods when there are few human preferences. The code and videos of our method are available at: https://sites.google.com/view/pearl-preference.

1. Introduction

Preference-based RL learns rewards from human judgments, but collecting labels for every task is costly. PEARL transfers source-task preferences to unlabeled target tasks and uses uncertainty-aware reward learning; experiments report strong transfer and better performance when human preferences are scarce.

1. Introduction

  • Reward engineering remains effortful, and agents can exploit crafted rewards in unexpected and potentially unsafe ways.Aligning crafted rewards with human values for sensorimotor skills also remains challenging in real-world applications.Source: §1 · p. 1
  • Preference-based RL still requires many human labels for new tasks, and labels collected for one task cannot be reused for others.This compounds the time and cost of training with human preferences.Source: §1 · p. 1
  • CPA aligns source and target trajectories with Gromov-Wasserstein distance, then uses their transport correspondence and source preferences to infer target-task labels.The framework applies robust reward learning to those labels before offline RL produces a policy.Source: §1 · p. 2, Figure 1 · p. 2
  • RRL models reward distributions to capture both average reward and uncertainty, addressing potentially incorrect labels transferred by CPA.The paper proposes this approach to support robust learning from transferred labels.Source: §1 · p. 2
  • PEARL's experiments on Meta-World and Robomimic report accurate zero-shot preference transfer and significantly better performance than current methods when human preferences are scarce.The introduction reports this result without giving a specific metric or value.Source: §1 · p. 2

2. Related Work

Prior work improves preference-feedback efficiency through exploration, pseudo-labeling, reward confidence, and reward-model changes, but new tasks still often require human labels. PEARL connects cross-task transfer to optimal transport and applies distributional modeling to rewards rather than features.

2. Related Work

  • Preference-feedback methods improve efficiency through unsupervised exploration, pseudo-labeling, uncertainty-guided exploration, Q-function signals, label smoothing, and conservative policy regularization.The cited methods include PEBBLE, SURF, RUNE, Meta-Reward-Net, and SEER.Source: §2 · p. 2
  • Meta-Reward-Net reduces adaptation costs, but still requires 1.5 million pretraining labels and new-task queries; PT and IPL still require hundreds of human labels.PEARL instead uses a small number of source-task labels to obtain extensive new-task preference labels.Source: §2 · p. 3
  • Prior OT work includes cross-task imitation and representation alignment, whereas PEARL directly aligns trajectory distributions to transfer preferences without representation-space alignment.The cited comparison states that representation-space alignment requires additional gradient computation.Source: §2 · p. 3
  • Distributional modeling represents features as Gaussian distributions in several fields, and prior preference-based RL work applies it to features; PEARL instead models reward distributions.The authors describe their reward-distribution modeling as the first of its kind in preference-based RL, to their knowledge.Source: §2 · p. 3

3. Problem Setting & Preliminaries

The setting transfers preferences between tasks that share an action space while allowing their states, dynamics, rewards, and discount factors to differ. The preliminaries define trajectory-pair preferences and introduce Gromov-Wasserstein distance as a relational basis for cross-domain alignment.

Problem Setting

  • Source and target tasks share an action space, but may differ in state space, transition function, reward function, and discount factor.Each task is modeled as an MDP.Source: §3 · p. 3
  • The transfer setting provides M source trajectories with pairwise preference labels and N target trajectories, then learns a target-task policy using transferred preferences.The stated goal is policy learning for the target task.Source: §3 · p. 3

Preference-based Reinforcement Learning

  • A trajectory segment is a sequence of states and actions; its pairwise label is 0, 1, or 0.5 for preference of the first, preference of the second, or equal preference.The preference predictor uses a Bradley-Terry model, and the reward parameters are optimized with cross-entropy before RL learns from labeled transitions.Source: §3 · pp. 3–4

Optimal Transport

  • Gromov-Wasserstein distance compares relational distance structures across domains, providing an alignment metric for source and target trajectories.The paper sets p to 2 in its definition; Figure 2 depicts alignment, transport correspondences, and target preference-label computation.Source: §3 · p. 4, Figure 2 · p. 4

4. Method

PEARL transfers preferences between tasks through optimal-transport trajectory alignment and models reward uncertainty to support robust learning without target-task human labels.

4.1. Cross-task Preference Alignment

  • CPA uses Gromov-Wasserstein optimal transport to match source and target trajectories, then weights source pair preferences by pair correspondence to infer target labels.The solved transport matrix represents trajectory correspondence; matched source trajectory pairs contribute more to transferred preferences.Source: §4.1 · pp. 4–5

4.2. Robust Reward Learning

  • RRL models rewards as Gaussian distributions with means and variances, trains preference prediction on mean and sampled rewards, and regularizes uncertainty to prevent collapse.The uncertainty regularizer aims to keep uncertainty at a level η.Source: §4.2 · p. 6

4.3. Practical Algorithm

  • The practical pipeline computes, normalizes, and binarizes CPA labels, trains RPT on them, then relabels offline data and trains an offline RL algorithm such as IQL.RPT implements distributional reward modeling based on PT.Source: §4.3 · p. 6

5. Experiments

Experiments evaluate PEARL on robotic manipulation tasks in zero-shot and limited-label settings, then test cost-function and label-noise variations. The comparisons include preference-transfer methods and baselines trained with scripted labels.

5. Experiments

  • The evaluation spans zero-shot task pairs, limited-data settings, and cost-function and noise tests on Meta-World and Robomimic manipulation tasks.Source: §5 · p. 6

5.2. Results of Zero-shot Preference Learning

  • 78.1% success across all tasks for RPT+CPA equals 93.2% of oracle performance, and PEARL exceeds PT+Sim while remaining comparable to scripted-label PT.CPA achieves 74.9% mean accuracy in computing preference labels across all tasks.Source: Table 1 · p. 7, §5.2 · p. 8

5.1. Compared Methods and Training Details

  • Button Press, Faucet Close, and Square-MH serve as source tasks; comparisons include PT, PT+Semi, PT+Sim, IPL, and CPA variants.Targets include Window Open, Door Close, Drawer Open, Sweep Into, Can-MH, and Lift-MH.Source: §5.1 · p. 7

5.3. Results of Few-shot Preference Learning

  • With fewer oracle labels, RPT+CPA significantly outperforms PT and surpasses Oracle PT on Window Open at 5–20 labels and Lift-MH at 0–20.The comparison uses the same number of scripted target-task labels for RPT+CPA and PT, while RPT+CPA additionally uses CPA labels.Source: Figure 4 · p. 8, §5.3 · pp. 8–9

5.4. Ablation Study

  • On Window Open, Cosine CPA reaches 91.0% label accuracy and 92.4% success, above scripted-label PT’s 89.2% success.CPA is reported as robust to either Euclidean or Cosine cost functions.Source: Table 2 · p. 8, §5.4 · p. 9
  • At higher label-noise levels, RPT significantly outperforms PT; noise is introduced by flipping 10%, 20%, or 30% of scripted labels.Figure 5 covers Sweep Into, Window Open, and Lift-MH.Source: Figure 5 · p. 9, §5.4 · p. 9

6. Conclusion

PEARL aligns source- and target-task trajectories with Gromov-Wasserstein optimal transport to transfer preference labels, then uses reward uncertainty to support robust learning. Experiments on Meta-World and Robomimic show accurate zero-shot preference transfer and stronger performance with few preferences, while the method has input-dimensionality and action-space constraints.

6. Conclusion

  • Cross-task Preference Alignment uses Gromov-Wasserstein distance and an optimal transport matrix to align trajectories and transfer preference labels from prior tasks.It requires only a small amount of source-task preference data, not extensive target-task human queries.Source: §6 · p. 9
  • Robust Reward Learning models reward uncertainty rather than scalar rewards to support robust reward-model learning from transferred labels.Source: §6 · p. 9
  • PEARL significantly surpasses current methods when preference labels are few, while experiments show accurate zero-shot transfer and more robust learning from noisy labels.The reported experiments cover robotic manipulation tasks in Meta-World and Robomimic.Source: §6 · p. 9
  • Optimal transport may process high-dimensional inputs slowly, and the algorithm's efficiency relies on source and target tasks sharing an action space.The paper identifies representation learning as a potential direction for addressing these constraints.Source: §6 · p. 9

A. Algorithm

  • The full algorithm clusters target trajectories, computes CPA labels, learns a reward model from them, and then trains a policy with offline RL.Source: A. Algorithm · p. 13

B. Preference Transformer

  • Preference Transformer models non-Markovian rewards with attention-weighted reward sums and optimizes its parameters from trajectory-pair preference data.Source: B. Preference Transformer · p. 14

C.1. Tasks

The experiments use nine robotic manipulation tasks: six Meta-World tasks and three Robomimic tasks. Their objectives range from button and fixture manipulation to object relocation and lifting.

C.1. Tasks

  • The button-press task manipulates a button at an arbitrarily placed initial position; faucet-close, window-open, door-close, and drawer-open tasks begin from randomly or arbitrarily chosen positions.Source: §C.1 · p. 14
  • The Can task repositions a can between containers, Sweep Into propels a ball into a cavity, Square places a square nut on a rod, and Lift raises a cube to a predefined height.Source: §C.1 · p. 14
  • Figure 6 covers nine robotic manipulation tasks, with panels (a–f) for Meta-World and (g–i) for Robomimic.Source: §C.1 · p. 14, Figure 6 · p. 15

C.2. Datasets

The datasets combine small source-task trajectory sets with task-specific target data. Meta-World source data use ground-truth and random policies, while Robomimic source and target data come from its Multi-Human offline dataset.

C.2. Datasets

  • Meta-World source datasets contain four trajectories per task: two from a ground-truth policy and two from a random policy, each 50 steps long.Source: §C.2 · p. 15
  • Meta-World target offline datasets use PEBBLE replay and feedback buffers, with task-specific budgets of 120,000 or 400,000 steps and 2,000–8,000 scripted labels.Window Open and Door Close use 120,000 steps and 2,000 labels; Drawer Open uses 400,000 steps and 4,000 labels; Sweep Into uses 400,000 steps and 8,000 labels.Source: §C.2 · p. 15
  • Robomimic source data draw from each task's Multi-Human offline dataset, collected by six operators across three proficiency levels, with 300 demonstrations overall.Each source set uses the best two trajectories and two random trajectories; the offline dataset also serves as target data.Source: §C.2 · p. 15

C.3. Implementation Details

The implementation removes goal information for Meta-World and derives CPA labels by clustering feedback-buffer trajectory segments. CPA accuracy is measured against ground-truth pairwise preferences; PT and IQL hyperparameters are listed separately.

C.3. Implementation Details

  • Meta-World states are made partially observable to remove goal information; feedback-buffer segments are clustered into two categories, with N = 4 and two sampled trajectories per category for CPA labeling.Source: §C.3 · p. 15
  • CPA accuracy is the percentage of trajectory-pair comparisons whose inferred labels match ground-truth preferences.Ncorrect counts matching pairs and Ntotal is the total number of pair comparisons.Source: §C.3 · pp. 15–16
  • The paper lists separate hyperparameter tables for Preference Transformer and IQL.Source: §C.3 · p. 16, Table 3 · p. 16, Table 4 · p. 16

D. Additional Results

Additional experiments examine sensitivity to three hyperparameters and the number of source-task trajectories. PEARL retains strong success rates across the tested settings, with larger α slightly reducing performance.

D. Additional Results

  • Figure 7 demonstrates that PEARL is robust to λ values of 0.001, 0.01, 0.1, and 1.λ balances the effects of mean and sampled rewards.Source: D. Additional Results · p. 16, Figure 7 · p. 17
  • Figure 8 shows high success rates for small α values, while larger α slightly reduces performance by diminishing Lce’s contribution to reward learning.The authors link the reduced contribution to lower reward-function accuracy.Source: D. Additional Results · p. 16, Figure 8 · p. 17
  • Figure 9 reports that the method is not sensitive to changes of η for the Window Open and Lift-MH tasks; the cited passage does not extend this finding to Sweep Into.Source: D. Additional Results · p. 17, Figure 9 · p. 17
  • PEARL is not sensitive to the number of source-task trajectories among the tested counts.The experiments evaluate M ∈ {4, 8, 16}; Table 5 reports success rate and CPA-label accuracy, with success-rate mean and standard deviation across five runs.Source: D. Additional Results · pp. 17–18, Table 5 · p. 18

E.1. How does the task similarity influence CPA?

The analyses associate greater task similarity with higher CPA accuracy, show that CPA can use incomplete pairwise preferences, and report higher standard deviation for PEARL than for PT and PT+Semi.

E.1. How does the task similarity influence CPA?

  • Lower reconstruction error and latent distance are associated with higher CPA accuracy; their Pearson correlations with CPA accuracy are −0.43 and −0.67, respectively.The authors report that PEARL’s performance diminishes with substantial domain gaps, such as D4RL Gym Locomotion.Source: §E.1 · p. 18, Table 6 · p. 18

E.2. Does CPA work without a full ranking of all trajectory pairs?

  • CPA can operate with sparse, incomplete pairwise preference information because its alignment matrix need not contain a full ranking of source-trajectory pairs.The authors describe checking whether all elements of the alignment matrix are zero when the most similar pairs lack explicit preference labels.Source: §E.2 · pp. 18–19

E.3. Standard Deviation of PEARL.

  • PEARL has higher standard deviation than PT and PT+Semi, with results varying across random seeds.The authors attribute this variability to reinforcement-learning environment stochasticity and PEARL’s zero-shot setting.Source: §E.3 · p. 19

Related papers

Loading 2306.03615v2…