PaperlayerView exact revision on arXiv

Source-linked AI summary

PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

Ruiqi Wang, Dezhong Zhao, Xuesu Xiao, Erdem Biyik, Byung-Cheol Min, Guohua Chen

arXiv:2610.31812v1cs.ROcs.AI

TL;DR

PbRL typically requires costly human preference labels for each new task, limiting its scalability. PrefO2O transfers preference structure from offline source-task data through a task-invariant latent space and confidence-weighted optimal transport. Across manipulation and locomotion benchmarks, it outperforms preference-transfer baselines and is competitive with or sometimes superior to methods trained with target-task supervision.

  • Problem

    Existing zero-shot preference transfer often aligns shallow trajectory similarities, which can be brittle under distribution shifts across dynamics, objectives, or state-action dimensions.

    Source: §I · p. 1

  • Method

    PrefO2O learns a task-invariant latent preference space from source tasks and uses OOD-aware optimal transport to generate confidence-weighted pseudo-preference supervision for online target-task learning.

    Source: §I · pp. 1–2

  • Results

    Across manipulation and locomotion benchmarks under intra- and cross-embodiment protocols, PrefO2O consistently outperforms preference-transfer baselines and is competitive with, and in several settings superior to, oracle RL and target-supervised PbRL.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The results support transferring relational, behavior-level contrasts underlying human preference judgments rather than the raw trajectory distribution itself.

    Source: §VI · p. 13

    Transfer assumes related source and target task families with shared behavioral structure; tasks whose preferences depend on entirely novel behavioral dimensions remain an open challenge.

    Source: §VI · p. 13

Abstract

from arXiv · show

Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for learning robot behaviors aligned with human intent. However, its scalability is fundamentally limited by the need to collect expensive human preference labels for every new task. In this work, we present PrefO2O, an offline-to-online latent preference transfer framework that enables zero-shot PbRL on unseen target tasks without requiring any human feedback. The core idea is to learn a task-invariant latent preference space from a small amount of offline preference data on source tasks, where transferable preference structures are disentangled from task-specific variations. This learned manifold allows PrefO2O to transfer preference structures and reward priors to new tasks under distribution shift, while confidence-weighted optimal transport alignment generates reliable pseudo-preference supervision for online reward adaptation. Extensive experiments on robotic manipulation and locomotion benchmarks demonstrate that PrefO2O substantially outperforms existing preference-transfer baselines and matches or exceeds PbRL methods trained with real human feedback, while generalizing across multi-to-one and one-to-one transfer settings as well as across different robot embodiments.

I. INTRODUCTION

PbRL learns rewards from human comparisons but typically requires new preference labels for every task, limiting its scalability. PrefO2O transfers shared preference structures learned from source tasks to unseen targets, where it provides reward priors and pseudo-preference supervision for online learning.

I. INTRODUCTION

  • Per-task collection of high-quality human preference labels is a scalability bottleneck for PbRL in real-world robotics.The paper notes that labels are typically collected from scratch for each new task.Source: §I · p. 1
  • Existing zero-shot preference transfer can be brittle under distribution shift because it aligns shallow trajectory-level similarities and offers limited guidance for target-task reward adaptation and policy improvement.The passage identifies differences in dynamics, objectives, and state-action dimensionality as challenging shifts.Source: §I · p. 1
  • PrefO2O learns a task-invariant latent preference space from offline data across source tasks to transfer preference priors and generate pseudo-preference supervision on unseen targets.The learned space abstracts task-specific variations; online alignment uses confidence-weighted pseudo-preference labels.Source: §I · pp. 1–2, Figure 1 · p. 2
  • PrefO2O jointly trains a trajectory encoder, preference encoder, and meta reward model on source-task preference data to represent behavior, capture shared preferences, and provide a transferable reward prior.The trajectory encoder maps potentially heterogeneous trajectories into compact representations.Source: §I · pp. 1–2

A. Preference-Based Reinforcement Learning

  • PbRL models comparative feedback as labels over trajectory pairs and learns a reward model from those relative judgments.Labels distinguish preference for either trajectory or indifference.Source: A. Preference-Based Reinforcement… · p. 2

B. Preference Transfer in PbRL

Preference transfer reuses source-task feedback to reduce or eliminate target-task annotation, but existing approaches face data, alignment, and representation limits. PrefO2O transfers preferences through a learned latent representation and uses optimal transport to establish soft correspondences for label propagation.

B. Preference Transfer in PbRL

  • Preference transfer aims to reuse source-task knowledge to reduce or eliminate feedback collection on new target tasks.The cited meta-learning approach still requires hundreds to thousands of target-task queries, whereas PrefO2O requires zero target-task human feedback.Source: B. Preference Transfer in PbRL · pp. 2–3
  • Existing preference-transfer methods can still require substantial labeled data, rely on raw state-action alignment that is sensitive to shifts, or assume shared state-action dimensions.The PEARL comparison also notes that label propagation alone does not learn a transferable representation for online adaptation.Source: B. Preference Transfer in PbRL · pp. 2–3
  • PrefO2O combines a task-invariant latent preference representation with OOD-aware optimal transport to support pseudo-preference generation under distribution shift and online reward and policy adaptation.The jointly trained trajectory encoder and meta reward model provide transferable structure for target-task learning.Source: B. Preference Transfer in PbRL · p. 3

C. Optimal Transport

  • Optimal transport produces a transport plan with soft correspondences between samples, which can propagate preference supervision from labeled to unlabeled trajectory pairs.PrefO2O performs Gromov–Wasserstein alignment over its jointly learned task-invariant preference representation.Source: C. Optimal Transport · p. 3

III. METHODOLOGY

PrefO2O uses offline source-task preferences to learn transferable representations and then performs online zero-shot preference transfer on an unlabeled target task. Its setting permits heterogeneous tasks within a related domain family, and its trajectory encoder separates task-specific input adaptation from shared representation learning.

A. Problem Setting and Assumptions

  • The target task has no human preference labels; target trajectory pairs are sampled from online policy rollouts for pseudo-preference generation.The assumed transfer scope is related task families with shared behavioral preference structure, while dynamics, objectives, and state-action dimensions may vary.Source: A. Problem Setting and Assumptions · p. 3

B. Overview

  • Offline training jointly learns a trajectory encoder, a preference VAE, and a meta reward model from source-task preference data; online, the encoder and reward model warm-start learning while the frozen VAE anchors OT alignment.Transport correspondences yield OOD-aware confidence-weighted pseudo-labels for reward learning and policy optimization.Source: B. Overview · pp. 3–4, Figure 2 · p. 4

C. Offline Latent Preference Learning

  • The trajectory encoder maps heterogeneous trajectories into compact behavioral representations, while the preference VAE models task-invariant preference structure over pairwise trajectory differences and the meta reward model predicts preference-aligned rewards.The components are jointly trained on source-task preference data.Source: B. Overview · p. 3, C. Offline Latent Preference Learning · p. 4

1) Trajectory Encoder:

  • Task-specific input adapters map state-action pairs to a common hidden dimension, and shared step encoding, temporal average pooling, and projection yield comparable trajectory-level representations.This separates accommodation of heterogeneous state-action dimensions from shared behavioral representation learning.Source: 1) Trajectory Encoder · p. 4

2) Preference VAE:

The preference VAE encodes pairwise trajectory differences into a structured latent preference space. Its training objectives promote regularity and cross-task alignment while preserving preference-discriminative information.

2) Preference VAE:

  • The preference encoder maps pairwise trajectory differences to stochastic latent preference vectors, and the decoder reconstructs those differences.The model is trained with complementary objectives.Source: 2) Preference VAE · p. 4
  • VAE regularization and moment matching promote a smooth, continuous latent manifold whose task-specific statistics align with global source-task statistics.The stated purpose is to support robust transfer during online learning.Source: 2) Preference VAE · p. 5
  • An auxiliary preference head is trained with cross-entropy so the latent space retains preference-discriminative information and preference directionality.The paper combines this discrimination objective with geometric regularization and cross-task distributional alignment.Source: 2) Preference VAE · p. 5

3) Meta Reward Model:

PrefO2O pairs a transferable reward prior with a structured latent preference space, then uses that space to generate target-task pseudo-preferences for online adaptation.

3) Meta Reward Model:

  • The meta reward model is trained on trajectory latents and aggregated source-task preferences, providing a transferable reward prior for target-task adaptation.Together with the preference VAE, it initializes target-task PbRL with both a preference-structured representation and a preference-informed reward model.Source: 3) Meta Reward Model · p. 5

4) Joint Training Objective:

  • Joint training sends all loss-term gradients through the trajectory encoder, making its representations preference-relevant and structurally transferable.The encoder sits upstream of both the preference VAE and meta reward model.Source: 4) Joint Training Objective · p. 5

D. Online Zero-Shot Preference Transfer

  • Online target learning freezes the preference encoder as a stable reference for OOD-aware optimal-transport alignment and confidence-weighted pseudo-preference supervision.The resulting labels guide continued reward adaptation and policy optimization.Source: D. Online Zero-Shot Preference Transfer · p. 5

1) Warm-Start Initialization:

  • Warm-starting transfers the trajectory encoder, meta reward model, and preference VAE; the VAE stays frozen while the trajectory encoder adapts to target dynamics.The input adapter is freshly initialized if the target state-action dimension matches no source task.Source: 1) Warm-Start Initialization · p. 5

2) Preference Alignment via Optimal Transport:

  • Optimal-transport alignment propagates source preference labels through latent-space correspondences to produce normalized soft labels for target pairs.The transport plan quantifies correspondence strength between source and target pairs in the learned latent preference space.Source: 2) Preference Alignment via Optimal… · p. 5

3) OOD-Aware Sampling and Reward Learning:

PrefO2O manages source–target discrepancy by scheduling target-pair sampling and weighting pseudo-labels according to their reliability.

3) OOD-Aware Sampling and Reward Learning:

  • The OT batch shifts from mostly in-distribution pairs early to include more OOD pairs later, using preference-VAE reconstruction error to classify them.This curriculum prioritizes manifold-aligned pairs with more reliable labels early, then broadens target-pair coverage.Source: 3) OOD-Aware Sampling and Reward… · p. 6
  • Each pseudo-label receives a confidence weight combining transport certainty and proximity to the frozen source preference manifold.The weight is high when labels agree on one outcome and the target encoding lies close to the source manifold.Source: 3) OOD-Aware Sampling and Reward… · p. 6
  • The confidence weights modulate reward adaptation through a weighted Bradley–Terry loss, with gradients backpropagated through the reward model and trajectory encoder.A sliding window of H recent state-action pairs converts trajectory-level rewards into stepwise signals, followed by standard SAC updates.Source: 3) OOD-Aware Sampling and Reward… · p. 6

IV. EXPERIMENTAL SETUP

The experiments assess zero-shot preference transfer across manipulation and locomotion tasks, varying source diversity and embodiment compatibility.

IV. EXPERIMENTAL SETUP

  • The study asks whether PrefO2O enables efficient zero-shot PbRL, generalizes across transfer settings, benefits from its components, captures transferable preferences, and can be deployed in real-world scenarios.These are framed as five research questions.Source: §IV · p. 6

1) Environments and Target Tasks:

  • Evaluation covers five Meta-World targets, two Robosuite targets, and Walker Walk from the DeepMind Control Suite.The tasks span manipulation and locomotion behaviors.Source: 1) Environments and Target Tasks · p. 6

2) Transfer Settings:

  • Transfer varies embodiment compatibility and source diversity, with multi-to-one as default and one-to-one testing transfer from a single source task.Cross-embodiment settings connect Meta-World and Robosuite; the one-to-one setting removes Lmom because cross-source statistics are unavailable.Source: 2) Transfer Settings · pp. 6–7

3) Offline Preference Data Collection:

  • Each offline source-task dataset uses mixed-quality trajectories and 100 human preference annotations collected from graduate students and staff with prior robotics exposure.Trajectory pairs are sampled from pools generated by policies of varying performance.Source: 3) Offline Preference Data Collection · p. 7

B. Baselines and Ablation Models

  • The study compares PrefO2O against baselines from three categories.The supplied setup identifies the categories but does not enumerate them in this passage.Source: B. Baselines and Ablation Models · p. 7

1) Cross-Domain Preference Transfer Methods: •

PrefO2O outperforms the supplied baselines across tasks, matching or exceeding oracle methods while transferring across source and embodiment settings.

1) Cross-Domain Preference Transfer Methods: •

  • The comparison includes PEARL, which aligns preferences via optimal transport in raw state-action space, and PrefFew, a meta-learning-based few-shot PbRL method given 100 target-task labels.PEARL is implemented with the online backbone and zero-padded inputs for cross-embodiment compatibility.Source: 1) Cross-Domain Preference Transfer… · p. 7

2) Foundation-Model-Based Feedback Methods: •

4) Evaluation:

  • Evaluation runs five seeds, reports manipulation success rate and locomotion episode return, and measures pseudo-label accuracy against a scripted teacher’s ground-truth-reward preferences.Ablations remove latent-space components, transferred reward priors, joint training, OOD sampling, or confidence weighting.Source: 4) Evaluation · p. 7, 3) Oracle Methods: • · p. 7

A. Overall Performance

  • PrefO2O surpasses all baselines across tasks and matches or exceeds oracle RL and PbRL methods, while achieving fast early-training coverage.Fig. 3 presents learning curves under intra- and cross-embodiment protocols.Source: Figure 3 · p. 8, A. Overall Performance · p. 8
  • PrefO2O retains the highest pseudo-label accuracy across benchmarks under intra-embodiment transfer, while PEARL and RL-VLM-F collapse to a small fraction of their intra-embodiment level under cross-embodiment transfer.Fig. 4 reports pseudo-label accuracy across three benchmarks.Source: Figure 4 · p. 8, A. Overall Performance · p. 8
  • PrefFew performs poorly in the source-limited setting of 100 labels per task across three source tasks, compared with its original formulation’s roughly 10,000 labels across ten tasks.The authors attribute its performance to reliance on meta-training and contrast it with PrefO2O’s transfer from a handful of source preferences and zero target labels.Source: A. Overall Performance · p. 8
  • Fig. 5 compares single-source curves with the multi-source reference; results show generalization across intra- and cross-embodiment, multi-source, and single-source regimes.Many single-source curves remain competitive, and some approach the multi-source reference.Source: Figure 5 · p. 9

B. Generalization Across Transfer Settings

PrefO2O transfers robustly across embodiment and source-diversity shifts; ablations indicate that its latent preference space and online OOD mechanisms support both adaptation speed and final performance.

B. Generalization Across Transfer Settings

  • Under cross-embodiment transfer, PrefO2O drops only 5 to 15 percentage points, while preference-transfer baselines’ relative rankings collapse.All transfer methods degrade when embodiment changes, but PrefO2O’s decline is marginal by comparison.Source: B. Generalization Across Transfer… · p. 9
  • With one source task, many transfer curves remain competitive and sometimes approach the multi-source reference, though performance depends on source-target semantic compatibility.Across four MetaWorld tasks, results also support generalization across intra- and cross-embodiment, multi-source, and single-source regimes.Source: B. Generalization Across Transfer… · p. 9

C. Ablation Study

  • Removing the latent preference space most consistently slows convergence, lowers final performance, and increases variance, especially on Drawer Close and Window Close.Raw-trajectory alignment exposes preferences to task-specific variation.Source: C. Ablation Study · pp. 9–10
  • Every ablation variant degrades in convergence speed, final performance, or both compared with full PrefO2O on four target tasks.The ablation figure compares six variants against the full method under intra-embodiment, multi-to-one transfer.Source: C. Ablation Study · p. 9, Figure 6 · p. 10
  • Removing joint training or the preference VAE causes slower early learning and a lower plateau, indicating that both shape transferable latent structure.Their contribution is geometric regularity and preference discrimination, rather than compactness alone.Source: C. Ablation Study · p. 10
  • Removing OOD sampling or weighting matters most on Stack Cube and Walker Walk, where sparse source coverage leaves target preferences underrepresented on the manifold.These mechanisms deprioritize and down-weight off-manifold pairs to keep online pseudo-supervision clean.Source: C. Ablation Study · p. 10

D. Latent Space Analysis

Latent-space analyses find task-invariant, preference-structured representations and meaningful OT correspondences; physical trials show transferred policies succeeding on both tested tasks, while the conclusion reports broader benchmark gains and a domain-family boundary.

D. Latent Space Analysis

  • Across intra- and cross-embodiment settings, latent representations mix tasks while separating preference labels, and their preference axis tracks interpretable behavioral improvements.The analyzed attributes are Approach Progress, Path Directness, and Motion Smoothness; the trend is strongest for Approach Progress and weaker for Motion Smoothness.Source: D. Latent Space Analysis · p. 11
  • For Sweep Into, OT matches target pairs to source pairs across tasks according to shared behavioral criteria, not task identity.Different target pairs match on different attributes, including approach efficiency or successful object engagement.Source: D. Latent Space Analysis · p. 11, Figure 9 · p. 12

E. Real-World Robot Validation

VI. CONCLUSION AND DISCUSSION

  • Across manipulation and locomotion benchmarks, PrefO2O outperforms preference-transfer baselines and is competitive with or exceeds target-supervised methods; it transfers effectively from one source task.The authors attribute gains to reliable latent-space correspondences and transferable initialization from the pretrained trajectory encoder and meta reward model.Source: §VI · pp. 12–13
  • Transfer to tasks whose preferences depend on behavioral dimensions absent from all source tasks remains an open challenge.The approach assumes source and target tasks belong to a related domain family with shared behavioral structure.Source: §VI · p. 13

APPENDIX A DETAILS OF OOD-AWARE CURRICULUM SAMPLING

The curriculum detects target pairs off the source preference manifold using a frozen VAE, then adjusts its ID/OOD balance and threshold over online training. It samples reliable ID pairs and novel, diverse OOD pairs to build each batch.

A. VAE Reconstruction-Error-Based OOD Detection

  • Per-sample VAE reconstruction error measures how far a target preference pair lies from the source manifold, while a source-side baseline distribution calibrates that measure.The score needs one frozen-VAE forward pass per sample, with negligible online overhead.Source: A. VAE Reconstruction-Error-Based OOD… · p. 14

B. Curriculum Schedule and Adaptive Threshold

  • The cosine schedule raises the OOD mixing weight from 0.2 to 1, while its coupled quantile threshold narrows the ID set as training proceeds.The schedule begins with an ID-heavy batch; the OOD pool becomes more meaningful as target encodings move toward the source manifold.Source: B. Curriculum Schedule and Adaptive… · p. 14

C. Diversity-Aware Sampling Within Each Set

  • Each batch first samples ID pairs closest to the source manifold, then selects OOD pairs for novelty and diversity relative to the sampled anchors.The OOD weight combines normalized reconstruction-error novelty with a penalty for redundancy near existing anchors.Source: C. Diversity-Aware Sampling Within… · p. 14

D. Procedure

The appendix reports additional learning curves, defines behavioral contrasts, and documents the procedure and real-robot setup. The additional curves show that PrefO2O’s reported trends extend to two further tasks, including locomotion.

D. Procedure

  • The sampling procedure computes replay-buffer reconstruction errors, partitions pairs by the current threshold, then draws a mixed batch using the curriculum weights.Its operations are O(|B|), except the OOD diversity term, which is O(nID) per OOD candidate; the appendix reports no measurable overhead at experimental batch sizes.Source: D. Procedure · p. 15

APPENDIX B ADDITIONAL BASELINE LEARNING CURVES

  • On Sweep Into and Walker Walk, the learning-curve trends mirror the main paper and extend beyond its displayed tasks, including to locomotion.The appendix states that the experimental protocol is identical to the main paper’s.Source: Appendix B · p. 15

APPENDIX C BEHAVIORAL ATTRIBUTE CONTRAST

  • Behavioral contrasts compare paired trajectory segments on approach progress, path directness, and motion smoothness, indicating which trajectory scores higher and by how much.Approach progress uses end-effector distance to the task-relevant object; directness compares displacement with path length, while smoothness scores negative average acceleration magnitude.Source: Appendix C · p. 15

A. Hardware and Cameras

  • The real-robot experiments use a 7-DoF Kinova Jaco2 arm, wrist-mounted and fixed-view cameras, and calibrated observations in the same low-dimensional format as simulation.Object and tray poses are estimated from calibrated camera streams.Source: Appendix D · p. 15, A. Hardware and Cameras · p. 15

B. Sim-to-Real Configuration

  • The sim-to-real configuration randomizes dynamics and sensing in Robosuite, then sends the policy’s simulation-matched actions to the arm with motions clipped to a safe tabletop region.Randomized factors include object mass, friction, joint damping, contact parameters, sensor noise, sampling-rate variation, and observation delay.Source: B. Sim-to-Real Configuration · p. 16

Related papers

Loading 2610.31812v1…