PaperlayerView exact revision on arXiv

Source-linked AI summary

ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations

Jiahui Zhang, Yusen Luo, Abrar Anwar, Sumedh Anand Sontakke, Joseph J Lim, Jesse Thomason, Erdem Biyik, Jesse Zhang

arXiv:2505.10911v2cs.RO

TL;DR

Robot learning commonly requires task-specific demonstrations or manually designed rewards, limiting scalable adaptation to new tasks. ReWiND learns a language-conditioned reward model and offline-RL policy from a small demonstration set, then fine-tunes on unseen variations with online rewards. It improves reward metrics by 23-74%, beats baselines by 2x in simulation, and improves real-world pretrained policies by 5x.

  • Problem

    Robot learning typically requires expert demonstrations or manually designed reward functions for each new task, making task-specific supervision expensive and time-consuming.

  • Method

    ReWiND learns a language-conditioned reward model from limited demonstrations and diverse augmented data, pre-trains a policy with offline RL, and fine-tunes it online for unseen task variations.

  • Results

    23-74% relative improvements over reward-learning baselines, 2x better simulation performance, and 5x improvement for real-world pretrained policies are reported.

  • Takeaways & Limitations

    ReWiND supports sample-efficient adaptation to unseen robot manipulation tasks using language instructions without additional per-task demonstrations.

  • Takeaways & Limitations

    Online fine-tuning of large policy architectures remains compute- and time-intensive, making real-world online learning difficult despite being feasible in simulation.

Abstract

from arXiv · show

We introduce ReWiND, a framework for learning robot manipulation tasks solely from language instructions without per-task demonstrations. Standard reinforcement learning (RL) and imitation learning methods require expert supervision through human-designed reward functions or demonstrations for every new task. In contrast, ReWiND starts from a small demonstration dataset to learn: (1) a data-efficient, language-conditioned reward function that labels the dataset with rewards, and (2) a language-conditioned policy pre-trained with offline RL using these rewards. Given an unseen task variation, ReWiND fine-tunes the pre-trained policy using the learned reward function, requiring minimal online interaction. We show that ReWiND's reward model generalizes effectively to unseen tasks, outperforming baselines by up to 2.4x in reward generalization and policy alignment metrics. Finally, we demonstrate that ReWiND enables sample-efficient adaptation to new tasks, beating baselines by 2x in simulation and improving real-world pretrained bimanual policies by 5x, taking a step towards scalable, real-world robot learning. See website at https://rewind-reward.github.io/.

1 Introduction

ReWiND teaches robots unseen task variations from a handful of demonstrations by combining language-conditioned reward learning, offline policy pre-training, and online fine-tuning. Its reward model targets dense feedback, generalization, and robustness, enabling strong reward and policy results.

  • Motivation and Framework: ReWiND uses a handful of grounding demonstrations to teach robots unseen task variations without collecting new task-specific demonstrations.It addresses the expense and time required for per-task demonstrations and the effort of manually designing reward functions.
  • Motivation and Framework: The framework first learns a language-conditioned reward model, pre-trains a language-conditioned policy with offline RL, and fine-tunes it online on new task variations.Deployment uses reward labels from the learned model on online interaction episodes.
  • Reward Model Design: ReWiND’s reward model is designed to provide dense feedback, generalize to unseen tasks, and remain robust to diverse robot behaviors.The model uses progress prediction, video rewind, pretrained vision-language encoders, and diverse Open-X robotics data.
  • Results: 23-74% relative improvements over reward learning baselines are achieved on metrics measuring reward density, generalization, and robustness.The paper also evaluates policy adaptation in simulation and on a real-world bimanual robot.
  • Results: ReWiND beats baselines by 2x in simulation and improves real-world pretrained policies by 5x on success-rate evaluations.These results concern adaptation to new manipulation tasks using the learned rewards and pretrained policy.

2 Related Works

Prior reward-learning approaches often require target-task demonstrations, extensive human feedback, ground-truth state information, or predefined policy priors. ReWiND instead learns a language-conditioned reward and policy from non-target-task data to address unseen task variations.

  • Reward Learning: Inverse RL and related reward-learning methods generally require new target-task demonstrations to reward unseen tasks.ReWiND instead trains a general language-conditioned reward function from an initial demonstration set.
  • Reward Learning: Human-feedback approaches reduce some supervision burdens but still require extensive feedback for each unseen task.Examples include comparisons, preference rankings, critiques, corrections, interventions, and language feedback.
  • Pre-trained Reward Models: LLM-generated reward approaches typically rely on ground-truth state information, whereas ReWiND uses task descriptions and policy-execution videos.This distinction is relevant to real-world reward generation, where ground-truth state information is difficult to obtain.
  • Vision-Language Rewards: RoboCLIP, LIV, VLC, VICTOR, and GVL also reward unseen manipulation tasks from language without sub-task annotations or target-task demonstrations.The paper reports that these baselines underperform ReWiND in its limited-data setting.
  • Policy Priors: Foundation Actor-Critic uses potential-based shaping rewards from a pretrained VLM but depends on predefined policy priors.ReWiND learns policy priors through offline RL on non-target tasks instead.

3 ReWiND: Learning Rewards Without New Demonstrations

ReWiND learns a language-conditioned reward model from limited demonstrations, then uses it to pre-train and adapt a policy for unseen language-specified tasks. Its design combines progress prediction, video and language augmentation, diverse data, and robustness-oriented architecture choices.

  • Problem formulation: ReWiND formulates unseen-task learning as a language-conditioned policy maximizing rewards predicted from observation sequences and instructions.The policy selects actions from images and language, while the reward model outputs per-timestep estimated rewards.
  • Policy learning: ReWiND pre-trains its policy with offline RL and fine-tunes it online for a new language-specified task using rewards from the learned model and a success bonus.Online learning assumes access to a success signal; automatic success detection is identified as future work.
  • Reward learning: The reward model is trained to predict per-frame progress, producing stable targets that translate directly into dense, normalized rewards.Mismatched observation sequences are assigned zero progress to improve robustness to incorrect video-instruction pairings.
  • Reward learning: ReWiND addresses generalization, rollout alignment, and input robustness through diverse Open-X data, video rewind, and instruction generation.Video rewind creates failure-like trajectories from successful demonstrations, while generated instructions broaden language variation.
  • Video and language augmentation: Video rewind reverses a subsequence after an intermediate timestep, training the reward model to decrease reward on trajectories that resemble failed execution.The remaining frames after the rewind point are unused, and the resulting sequences are intended to mimic failures encountered during online reinforcement learning.
  • Reward model architecture: The reward model combines frozen vision and language encoders with a lightweight cross-modal sequential aggregator using positional information only on the first image.These architectural choices are designed to support generalization from the limited demonstration set without allowing frame position to directly reveal progress.

4 Experiments

ReWiND’s experiments evaluate reward quality, sample-efficient learning of unseen tasks, and the effects of its design choices. Across simulation and real-robot settings, ReWiND improves reward alignment, robustness, and policy success, while ablations identify video rewinding, instruction generation, dataset composition, and positional embeddings as important design factors.

  • Q1: Rewards: ReWiND produces clearer unseen-task video–instruction separation than baselines in confusion-matrix evaluation.Its generalization remains strong even without OXE data, attributed to architectural choices including first-frame positional encodings and feature design.
  • Q1: Rewards: 30% relative improvement in Pearson r and 27% in Spearman ρ over VLC measure ReWiND’s stronger progress alignment on unseen demonstrations.Pearson correlation measures agreement with temporal progress, while Spearman correlation captures monotonicity regardless of linearity.
  • Q1: Rewards: 74% relative improvement in reward order and 58% in reward difference over LIV-FT show stronger ranking of failure, near-success, and success rollouts.The evaluation uses Spearman’s ρ for ranking alignment and compares reward gaps between adjacent rollout-quality levels.
  • Q1: Rewards: ReWiND achieves average instruction correlation 0.74, 23% better than VLC, with near-zero variance across varied language inputs.Higher output variance indicates lower robustness; ReWiND maintains this result even without OXE training.
  • Q2: Policy Learning: 79% IQM success after 100k steps on eight unseen MetaWorld tasks is a 97.5% improvement over VLC, the best baseline.Sparse-reward fine-tuning and pre-training without fine-tuning achieve near-zero success, while ReWiND is also more sample-efficient before 100k steps.
  • Q2: Policy Learning: In five real-world tasks, ReWiND raises average success from 12% to 68%, a 5× improvement, and outperforms VLC by 6.7×.The experiments use five reward-model demonstrations and ten policy demonstrations; one hour of parallelized real-world RL corresponds to 50k environment steps.
  • Q3: Ablations: Removing OXE data harms unseen-task reward alignment and input robustness, whereas removing target-environment data harms demonstration alignment and rollout discrimination.Target-environment removal retains strong input robustness because of OXE diversity.
  • Q3: Ablations: Removing video rewinding degrades rollout ranking and reduces policy learning performance by 33%, while removing instruction generation reduces language-input robustness.These ablations support the roles of rewinding for failure discrimination and generated instructions for varied-language robustness.

5 Limitations

ReWiND’s current limitations concern policy strength, online fine-tuning, reward supervision, embedding coverage, environment resets, and success detection. These constraints limit autonomy, adaptation to difficult tasks, and the handling of certain trajectories or robotics settings.

  • Initial Policy Performance: Stronger policy architectures may improve ReWiND’s performance, because initial zero-shot performance strongly predicts real-world learning on new tasks.A policy confidently performing the wrong task was not helped by ReWiND, partly because KL-constrained learning makes unlearning poor behaviors slow.
  • Initial Policy Performance: Online fine-tuning of expressive policy models remains an open challenge because it requires substantial compute and real-world time.The paper reports that such fine-tuning is feasible in simulation but difficult for real-world online learning with rewards.
  • Reward Objective: Video-progress reward labeling cannot explicitly incorporate collected failure trajectories without labels for their final task progress.The paper suggests using failures as negatives with zero progress or adding alternative objectives, accurate VLMs, or human progress labels.
  • Reward Analysis: Frozen vision-language embeddings can underfit robotics-specific tasks, particularly when pretraining lacks bimanual data or partial-occlusion views.The paper gives dish scrubbing as an example where ReWiND performs poorly despite related linguistic tasks in Open-X.
  • Resets: ReWiND currently requires human environment resets, preventing fully autonomous learning and leaving real-world reset handling unresolved.Reset-free RL is identified as a promising direction, but human resets remain a roadblock to autonomy.
  • Success Detection: The method also requires success detection for reward bonuses and terminating rollouts after successful task completion.This requirement is tied to the use of a success bonus to compensate for noisy rewards and imperfect reward-model success detection.

A.1.1 Open-X Dataset

The Open-X subset supplies task-oriented manipulation trajectories for reward-model training, while ReWiND combines pretrained language and image representations with a sequential transformer reward architecture. The algorithm trains rewards, relabels demonstrations for offline policy pretraining, and then applies online RL to new tasks.

  • Open-X Dataset: The Open-X subset contains task-oriented manipulation trajectories selected from multiple robotics datasets for reward-model training.The listed sources include Bridge-V2, BC-Z, Fractal, CLVR Jaco Play, DROID, and other manipulation datasets.
  • Reward Function: ReWiND’s reward model encodes language with ALL-MINILM-L12-V2 and images with DINO-V2 before projecting both modalities to 512 dimensions.The projected embeddings are processed as tokens by a cross-modal sequential aggregator, which produces per-image embeddings for reward prediction.
  • Reward Function: The reward-model training procedure uses demonstration and Open-X data, augmented instruction labels, video clips, and progress optimization.The algorithm specifies reward-model training before offline policy pretraining and online learning for new tasks.
  • Reward Function: The aggregator is a causally masked transformer with 4 layers and 8 attention heads per layer.Training uses separate step counts for Meta-World and real-world experiments, with batches mixing Open-X and target-environment data.

A.1.3 Policy Training

ReWiND conditions its policy on frozen image-language embeddings and proprioception, pretrains with offline IQL, and fine-tunes with custom SAC initialized from the pretrained policy. Real-robot online RL required environment-specific design choices because standard offline-online methods performed poorly out of the box.

  • Policy Input: The policy receives frozen DINO-v2 image embeddings, ALL-MINILM-L12-V2 language embeddings, and proprioceptive information.The image embeddings have 768 dimensions, while the language embeddings have size 384.
  • Offline RL: Offline policy pretraining uses Implicit Q Learning on in-distribution dataset transitions.IQL avoids sampling next actions from a policy and uses expectile regression to estimate values from dataset actions.
  • Online RL: Online adaptation uses custom SAC initialized with the offline policy, Q functions, and target Q functions.The implementation includes multiple critics, normalization, increased critic updates, and warm-starting with the frozen policy.
  • Online RL: The policy-training implementation also considers sampling offline data during online RL and removing SAC’s entropy term from the target critic.These choices are listed as possible components of the environment-specific online-RL setup.
  • Online RL: Real-robot offline-online methods did not work well by default, so ReWiND uses per-environment online-RL design decisions.The paper attributes this difficulty either to task changes during fine-tuning or to additional challenges of real-robot RL.

B MetaWorld Experiments

The MetaWorld experiments train reward models and policies on 20 manually selected MT50 tasks, evaluate rewards on 17 other tasks, and fine-tune policies on 8 target tasks. The setup omits goal-state information and uses a 128-step horizon to resemble real-world learning.

  • Training/Eval Task Selection: The MetaWorld training set contains 20 manually selected MT50 tasks used for reward-model training and policy pretraining.The tasks span manipulation behaviors including button pressing, coffee pulling, door opening, pick-and-place, pushing, reaching, and window opening.
  • Training/Eval Task Selection: Reward-model evaluation uses 17 additional MT50 tasks, while downstream policy fine-tuning uses 8 selected target tasks.The 8 downstream tasks include Window-Close, Reach-Wall, Faucet-Close, Coffee-Button, Door-Lock, and Sweep-into.
  • Environment Details: The simulation uses third-person images, 4-dimensional proprioception, and a 4-dimensional relative action space.The action dimensions represent Δx, Δy, Δz, and gripper control.
  • Environment Details: MetaWorld experiments omit goal or ground-truth state information and terminate episodes on success within a 128-step horizon.The setup is intended to mimic real-world robot learning rather than provide privileged state information.

B.2 Training Details

Training combines simulated MetaWorld tasks with a low-cost real-world bimanual setup, using demonstrations, visual preprocessing, and offline-to-online RL procedures.

  • 20 MetaWorld tasks are selected from MT-50, with each task containing one human annotation, four augmented annotations, and five optimal demonstrations.
  • The real-world platform uses two Koch v1.1 arms with top-down and side cameras, while two additional leader arms support data collection.
  • MetaWorld images are rendered at 640x480, center-cropped to 224x224, and embedded with a DINOv2 encoder.
  • The MetaWorld policy is pretrained with IQL for 100K steps using a 0.001 learning rate, discount γ = 0.99, and three-layer MLP networks.
  • Online MetaWorld RL uses ten critics, samples two per update, applies LayerNorm, and warm-starts learning for 4000 steps.
  • The four-arm Koch setup costs approximately $1000 and uses RGB observations from Logitech top and side cameras with 30Hz absolute joint-position control.

C.2 Real Robot Training Details

Real-robot training augments language instructions, uses action-chunked Transformer policies, and regularizes online learning toward pretrained behavior for safety and stability.

  • A small instruction-tuned Mistral-7B-Instruct-v0.3 model generates nine additional instructions per task for language augmentation.
  • The real-robot dataset contains 15 training tasks with five trajectories and ten annotations each, evaluated on five unrelated random tasks.
  • The real-world policy predicts 60 actions, corresponding to two seconds, with a Transformer and temporal ensembling for more stable bimanual control.
  • The policy is pretrained offline for 20K steps with IQL and AWR, uses five critics, and warm-starts online RL for 3000 steps.
  • Online real-world training runs for 50k environment steps, taking approximately one hour through threaded rollout and policy-training parallelization.
  • To avoid unsafe random actions, online updates regularize against the pretrained policy distribution rather than a maximum-entropy prior.

C.3 Real Robot Tasks

Real-robot tasks test spatial, visual, and semantic generalization, while evaluation analyses compare reward alignment and downstream success across tasks.

  • C.3 Real Robot Tasks: The Koch dataset contains 20 tasks with ten demonstrations per task, using five demonstrations for reward training and ten for policy training.
  • C.3 Real Robot Tasks: The five online tasks include two in-distribution tasks and three tasks requiring visual, spatial, or semantic generalization.
  • C.3 Real Robot Tasks: Training and evaluation tasks are selected to assess spatial, visual, and language generalization.
  • C.3 Real Robot Tasks: Evaluation includes separating cups, folding a towel, opening a cluttered trash bin, placing a cup farther from training locations, and interpreting “fruit-colored” objects.
  • C.3 Real Robot Tasks: On MetaWorld, ReWiND produces the clearest diagonal-heavy reward confusion matrices among the compared models.
  • C.3 Real Robot Tasks: ReWiND achieves higher average success rates than baselines with the same number of MetaWorld timesteps and is generally more sample-efficient.

D.3 Real-World Reward Analysis

Real-world reward analysis evaluates rank correlation and video-language alignment on unseen tasks, where ReWiND compares favorably with alternative reward models.

  • ReWiND achieves the highest Spearman’s ρ and Pearson’s r among the evaluated real-world reward models.
  • Table 4 compares reward models on real-world unseen tasks using Spearman’s rank correlation ρ and Pearson’s rank correlation r.
  • On five unseen tasks, ReWiND shows the best alignment between paired videos and language instructions.
  • ReWiND also has the best alignment on training tasks, while LIV, GVL, and RoboCLIP are zero-shot models for these tasks.
Loading 2505.10911v2…