PaperlayerView exact revision on arXiv

Source-linked AI summary

GRAPE: Generalizing Robot Policy via Preference Alignment

Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, Huaxiu Yao

arXiv:2411.19309v2cs.ROcs.CVcs.LG

TL;DR

VLAs have limited generalizability and adaptability because they primarily imitate successful expert demonstrations whose objectives may be ambiguous. GRAPE aligns policies through trajectory-level preferences, stage-wise VLM-guided costs, and rewards from external objectives, self-evaluation, and task success. It improves success rates on in-domain and unseen tasks while supporting safety- and efficiency-oriented alignment.

  • Problem

    VLAs generalize poorly to unseen settings because behavior cloning from successful rollouts does not model holistic task goals or failure patterns, while demonstrations may encode unclear objectives.

  • Method

    GRAPE performs trajectory-wise preference optimization using successful and failed trials, with VLM-decomposed stages, keypoints, customized costs, self-evaluation, and task-success indicators.

  • Results

    GRAPE improves success rates on in-domain and unseen manipulation tasks by 51.79% and 58.20%, respectively.

  • Takeaways & Limitations

    GRAPE supports flexible alignment of VLA policies with task completion, safety, and efficiency objectives across real-world and simulated tasks.

  • Takeaways & Limitations

    The formulation assumes a step-wise policy under the MDP assumption, although the authors state it can be adapted to non-MDP histories and diffusion policies.

Abstract

from arXiv · show

Despite the recent advancements of vision-language-action (VLA) models on a variety of robotics tasks, they suffer from critical issues such as poor generalizability to unseen tasks, due to their reliance on behavior cloning exclusively from successful rollouts. Furthermore, they are typically fine-tuned to replicate demonstrations collected by experts under different settings, thus introducing distribution bias and limiting their adaptability to diverse manipulation objectives, such as efficiency, safety, and task completion. To bridge this gap, we introduce GRAPE: Generalizing Robot Policy via Preference Alignment. Specifically, GRAPE aligns VLAs on a trajectory level and implicitly models reward from both successful and failure trials to boost generalizability to diverse tasks. Moreover, GRAPE breaks down complex manipulation tasks to independent stages and automatically guides preference modeling through customized spatiotemporal constraints with keypoints proposed by a large vision-language model. Notably, these constraints are flexible and can be customized to align the model with varying objectives, such as safety, efficiency, or task success. We evaluate GRAPE across a diverse array of tasks in both real-world and simulated environments. Experimental results demonstrate that GRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 58.20%, respectively. Additionally, GRAPE can be aligned with various objectives, such as safety and efficiency, reducing collision rates by 37.44% and rollout step-length by 11.15%, respectively. All code, models, and data are available at https://grape-vla.github.io/

1. Introduction

VLAs generalize poorly because supervised fine-tuning imitates successful expert behavior without modeling task goals, failures, or explicitly defined objectives. GRAPE addresses this with trajectory-level preference optimization and customizable stage-wise constraints, improving task success and alignment with safety and efficiency objectives.

  • Motivation: VLAs struggle to generalize across new environments, objects, tasks, and semantic contexts because SFT imitates successful rollouts without learning holistic task goals or failure patterns.Behavior cloning from successful demonstrations is identified as a significant contributor to limited generalizability.
  • Motivation: Uncurated demonstrations embed unclear values such as task completion, safety, and cost-efficiency, which can confuse SFT and produce suboptimal trajectories.Hierarchical objective-solving approaches add inference overhead and lack scalability.
  • Approach: GRAPE aligns VLA policies at the trajectory level by implicitly modeling reward from both successful and failed trials.This preference-based formulation is intended to improve generalizability to diverse tasks.
  • Approach: GRAPE decomposes complex manipulation into stages and uses VLM-proposed keypoints with customizable spatiotemporal constraints for different objectives.The constraints can target task completion, robot-interaction safety, or cost-efficiency.
  • Results: 51.79% and 58.20%: GRAPE improves success rates on in-domain and unseen manipulation tasks, respectively.The evaluation spans real-world tasks and two simulated environments.

2. Generalizing Robot Policy via Preference Alignment

GRAPE replaces exclusive behavior cloning with trajectory-wise preference optimization and automatically generated, objective-specific trajectory scores. It decomposes tasks into stages, ranks sampled trajectories using external, self-evaluated, and success rewards, and iteratively updates the VLA policy.

  • Preliminaries: The preliminaries formulate VLA inference as iterated observation-action interaction over a trajectory and identify SFT’s expert-trajectory memorization as a source of poor generalization.The approach assumes a step-wise MDP policy but is described as adaptable to non-MDP histories and diffusion policies.
  • TPO: TPO globally aligns policies with trajectory preferences while learning from both successful and failed rollouts through step-wise state-action likelihoods.The objective backpropagates through all state-action pairs and is designed to improve generalizability.
  • GCPG: GRAPE addresses the annotation cost of arbitrary preferences by automatically generating trajectory rankings for task completion, safety, and efficiency.GCPG is introduced because manual preference synthesis requires human expertise and lengthy annotation.
  • GCPG: A VLM decomposes trajectories into temporal stages, while keypoints and LLM-generated stage costs quantify alignment with customized objectives.Lower stage cost indicates better objective compliance, and stage costs are aggregated into a holistic trajectory evaluation.
  • GCPG: The GCPG reward combines external objective-aligned cost, policy self-evaluation, and a binary task-success indicator.The self-evaluated score is the policy’s trajectory log-likelihood, while the success indicator records whether the task is completed.
  • Iterative Optimization: At each iteration, GRAPE samples trajectories, scores and ranks them, pairs top and bottom trajectories, and updates the same policy with TPO.The process repeats for K iterations to obtain a policy aligned with the target objective.

3. Experiment

GRAPE is evaluated against supervised and preference-tuned VLA baselines across simulated and real-world tasks, including multiple generalization settings. Across these evaluations, trajectory-wise preference optimization improves performance, supports iterative gains, and enables alignment with safety and efficiency objectives.

  • 3.2. Evaluation in Simulation Environment: GRAPE outperforms Octo-SFT and OpenVLA-SFT by averages of 131.72% and 46.10% in Simpler-Env, and 8.53% and 7.36% in LIBERO.Simpler-Env measures in-domain, subject, physical, and semantic generalization, while LIBERO evaluates four in-domain task types.
  • 3.2. Evaluation in Simulation Environment: GRAPE improves over OpenVLA-DPO by an average of 33.14% across both simulated environments, supporting trajectory-level learning from successful and failed trials.The comparison is attributed to global trajectory-level alignment without low-level step-wise noise.
  • 3.5. Analysis of Iterative Preference Optimization: Full reward-score preference ranking outperforms random success-based selection, while self-evaluated, external objective-aligned, and success-indicator components each contribute to performance.The components respectively support generation-probability robustness, customized behaviors such as safety and efficiency, and prioritization of successful trajectories.
  • 3.5. Analysis of Iterative Preference Optimization: GRAPE progressively improves across preference-optimization iterations, with gains of 17.5%, 9.0%, 15.0%, and 21.0% for in-domain, subject, physical, and semantic evaluations.The magnitude of improvement diminishes over time as the model approaches convergence.
  • 3.6.2. CASE STUDY: GRAPE adapts to safety and efficiency objectives by changing multi-stage cost functions, improving the targeted metric while maintaining comparable success rates to OpenVLA-SFT.A safety-aligned policy avoids an obstacle while completing a pickup-and-place task, unlike task-completion-only alignment.

4. Related Works

Prior VLA approaches use hierarchical planning or direct action generation, but typically rely on successful-rollout behavior cloning, limiting generalization; GRAPE instead aligns policies at the trajectory level through trial and error.

  • Vision-Language-Action Models: Hierarchical methods use language or vision-language models for high-level plans before relying on low-level controllers for local trajectories.
  • Vision-Language-Action Models: OpenVLA discretizes actions into tokens to preserve autoregressive decoding, but this can introduce errors.
  • Vision-Language-Action Models: Most VLA models are supervised-trained on successful rollouts via behavior cloning, which can generalize poorly to unseen manipulation tasks.
  • Reinforcement Learning and Preference Optimization: GRAPE addresses these limitations by aligning VLA policies at the trajectory level through trial and error rather than relying only on successful demonstrations.
  • Reinforcement Learning and Preference Optimization: Reinforcement-learning post-alignment is difficult because manipulation objectives are diverse, preference annotation is lengthy, and imperfect reward differentiation can hinder optimization.

5. Conclusion

GRAPE aligns VLA policies with trajectory-level preferences learned from successful and failed trials, while customized stage constraints support task completion, safety, and efficiency objectives. The conclusion reports improved success, lower collision rates, and shorter average trajectories.

  • GRAPE aligns VLA policies on a trajectory level and learns from both successful and failed trials to improve generalizability.
  • Customized cost generation uses VLM-produced stage information and stage points to segment complex trajectories for more precise analysis.
  • Task-specific collision and path constraints provide operational costs for safety and efficiency requirements.
  • Iterative preference optimization samples multiple trajectories per task and automatically labels them with the GCPG reward for preference selection.
  • TPO training uses LoRA and AdamW with a learning rate of 2 × 10^-5, batch size 16, and one training epoch.

B. Detail Experiment Datasets

The experiments use separate supervised fine-tuning and trajectory-preference datasets across real-world, Simpler-Env, and LIBERO settings. Real-world data cover pick-and-place, button-pressing, and knock-down tasks, while simulated datasets contain task-specific sampled trajectories.

  • The study distinguishes datasets for supervised fine-tuning from those used for trajectory preference alignment.
  • The real-world SFT collection uses a Franka arm with a Robotiq gripper in a consistent experimental environment.
  • Real-world SFT data include 220 pick-and-place instances, 50 button-pressing instances with testing-time background interference, and 50 knock-down instances.
  • The real-world TPO dataset samples 15 tasks five times each, yielding 75 entries and a selected preference dataset of 30 trajectories.
  • Simpler-Env uses 100 SFT trajectories and 80 TPO trajectories, while LIBERO reuses OpenVLA-SFT and forms a 20-trajectory TPO dataset across four tasks.

C.1.1. REAL-WORLD EXPERIMENT SETUP

The real-world setup uses a Franka arm with a Robotiq gripper and evaluates cross-task generalization against multiple baseline models. The gripper replacement addressed hardware limitations that caused inefficient completion and high failure rates.

  • Replacing the original Franka gripper with a longer Robotiq gripper improved grip and flexibility for the evaluated tasks.
  • The original gripper was too short for some tasks, resulting in inefficient completion and a high failure rate.
  • The experiment evaluates OpenVLA’s cross-task generalization under GRAPE against several baseline models using a supervised-fine-tuned model built from real-scene rollouts.

C.1.2. REAL-WORLD TASKS

The real-world evaluation spans five generalization scenarios across 30 tasks, with each task attempted ten times under matched testing conditions.

  • Generalization Scenarios: The real-world suite covers visual, subject, action, semantics, and language-grounding generalization.The scenarios include 8 visual, 4 subject, 7 action, 4 semantic, and 3 language-grounding tasks.
  • Evaluation Setup: 30 tasks were evaluated with ten attempts each, totaling 300 executions under matched starting positions, image resolution, and object placements.Task success used task-specific criteria, including accurate placement, successful knock-down, or button pressing.
  • Evaluation Metrics: Table 3 compares GRAPE models across in-domain and three generalization evaluations using grasp rate and overall success rate.The table evaluates performance across different iteration rounds.

C.2. Simulation Experiments

Simulation experiments use Simpler-Env and LIBERO to test in-domain performance and multiple forms of generalization across object, physical, semantic, spatial, goal, and long-horizon settings.

  • Simulation Environments: Simpler-Env provides simulated manipulation environments designed to approximate real-world scenarios while reducing the cost and difficulty of physical testing.The benchmark is presented as a practical alternative to real-world evaluation, which is costly, time-consuming, and difficult to replicate.
  • Simpler-Env: Simpler-Env evaluates four in-domain tasks plus subject, physical, and semantic generalization tasks.Subject generalization changes objects, physical generalization changes object sizes or collision boxes, and semantic generalization modifies instructions while preserving meaning.
  • Task Examples: The simulated tasks include object placement, stacking, drawer and cabinet manipulation, stove interaction, and multi-object arrangement.Examples include putting a carrot on a plate, stacking cubes, opening a drawer, and placing a frying pan on a stove.
  • LIBERO: LIBERO evaluates Spatial, Object, Goal, and LIBERO-10 suites covering spatial layouts, object types, task goals, and long-horizon combinations.The suites use distinct distribution shifts in object arrangements, objects, goals, or combinations of these factors.

D. Additional Real-World and Simulation Results

Additional results compare Octo-SFT, OpenVLA-SFT, and GRAPE across in-domain and several generalization settings using detailed task-level evaluations.

  • Cross-Model Results: Additional tables report comparisons among Octo-SFT, OpenVLA-SFT, and GRAPE across in-domain, subject, physical, and semantic generalization tasks.The evaluations include grasp percentages and success rates for each task.

E.1. Case Study of Real-World Generation Tasks

The case studies illustrate GRAPE’s real-world task coverage and its customized multi-stage cost functions for task completion, efficiency, and safety alignment.

  • Real-World Generation Tasks: GRAPE’s real-world evaluation includes in-domain, subject, visual, action, semantic, language, and safety task illustrations.The figures show initial and final states across the corresponding task categories, while Table 4 reports detailed results.
  • Multi-Stage Cost Functions: Multi-stage cost functions decompose manipulation tasks into stages such as grasping, moving, and dropping before assigning objective-specific constraints.The pipeline uses task stages and keypoints to construct costs for different alignment objectives.
  • Task Completion Alignment: Task-completion alignment measures distances between the end-effector and target keypoints across grasp, transport, and drop stages.Examples use grape and black-bowl centers as keypoints for stage-specific target costs.
  • Cost-Efficiency Alignment: Cost-efficiency alignment penalizes collisions and inadequate clearance around obstacles during staged manipulation.The collision cost increases when the end-effector approaches obstacles within a specified threshold.
  • Safety Alignment: Safety alignment encodes approach-direction and path constraints, such as approaching objects from above and minimizing distance-based path costs.These constraints are applied separately across grasping, transport, and dropping stages.
Loading 2411.19309v2…