PaperlayerView exact revision on arXiv

Source-linked AI summary

OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models

Xule Gao, Xi Wang, Zehua Zang, Rui Wang, Changwen Zheng, Chuxiong Sun

arXiv:2610.12914v1cs.AI

TL;DR

VLA policies trained by supervised imitation tend to follow demonstration-specific action paths, while online reinforcement learning requires costly environment rollouts. OGPO relabels offline trajectories around task-progress key states and uses goal-conditioned rewards; it consistently improves over SFT on manipulation benchmarks and strengthens robustness and recovery under distribution shifts and execution-time disturbances.

  • Problem

    VLA policies adapted through step-wise supervised imitation tend to reproduce demonstration-specific paths, while reinforcement-learning post-training typically requires costly environment rollouts.

    Source: §1 · pp. 1–2

  • Method

    OGPO adaptively relabels offline trajectories with task-progress key states and uses goal-conditioned process rewards to optimize policies while retaining SFT as an action prior.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    OGPO consistently improves over SFT on LIBERO and MetaWorld and strengthens robustness and recovery on LIBERO-Plus and LIBERO-ReAct; on LIBERO, average success rises from 77.1% to 97.1%.

    Source: §1 · p. 2, §4.2 · pp. 7–8

  • Takeaways & Limitations

    Offline demonstrations can support learning to recover toward task-progress key states without additional online interaction.

    Source: §1 · p. 2, §5 · p. 9

    OGPO’s gripper-state-based segmentation may not generalize to tasks whose progress is not naturally characterized by gripper-state events.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-language-action (VLA) models have shown strong promise in robotic manipulation, yet their post-training still predominantly relies on trajectory-level supervised imitation over robot demonstrations. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration-specific action paths and providing limited guidance for recovering toward task-progress states once execution deviates from demonstrated trajectories. To address this limitation, we propose OGPO, an Offline Goal-conditioned Policy Optimization framework for recoverable VLA post-training. Instead of treating offline demonstrations merely as step-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards to optimize the policy for reaching these task-progress states. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off-demonstration states without requiring additional online interaction. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine-tuning. Moreover, evaluations on LIBERO-Plus and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal-directed recoverability under distribution shifts and recoverable off-demonstration states.

1 Introduction

Trajectory-level SFT reproduces demonstration actions but offers limited guidance for recovering after execution deviates, while online RL can be costly to scale. OGPO uses offline key-state reachability learning, and evaluations report improved task performance, robustness, and recoverability.

Motivation

  • Trajectory-level SFT ties policies to demonstration-specific action paths and provides limited guidance for recovery toward task-progress states after deviations.Long-horizon execution deviations can lead to compounding errors; the paper illustrates SFT failure under perturbation in Figure 1.Source: §1 · p. 1, Figure 1 · p. 2
  • Existing RL post-training can optimize goal-directed behavior, but typically requires costly environment rollouts and has limited sample efficiency.These costs make such methods difficult to scale for real-world robotic post-training.Source: §1 · p. 2

OGPO

  • OGPO adaptively relabels demonstrations with task-progress key states and uses goal-conditioned process rewards to learn reachability beyond step-wise action imitation.The framework retains SFT as an action prior while avoiding additional online interaction.Source: §1 · p. 2

Evaluation

  • OGPO consistently improves over SFT on LIBERO and MetaWorld, with stronger robustness and goal-directed recoverability on LIBERO-Plus and LIBERO-ReAct.LIBERO-ReAct evaluates mid-execution perturbations; the reported gains include long-horizon and difficult tasks.Source: §1 · p. 2

2 Related Works

Prior VLA post-training mainly uses SFT or RL: SFT imitates expert actions, while RL can optimize task rewards but often depends on costly interaction. OGPO distinguishes itself by combining SFT regularization with offline goal-conditioned RL.

2.1 Supervised Fine-tuning for VLA Policies

  • SFT adapts VLA policies by predicting expert actions from demonstrations, but remains constrained by dataset coverage and offers limited support for unseen states, perturbations, and long-horizon errors.The section describes OpenVLA, Octo, π0, π0.5, and OpenVLA-OFT as examples of SFT-based approaches.Source: §2.1 · p. 3

2.2 Reinforcement Learning for VLA Post-training

  • RL post-training uses task-level rewards to move beyond expert behavior, but commonly relies on online rollouts, real-world interaction, or simulator-based evaluation.The section reviews online RL, policy-gradient, residual RL, distillation, and simulator- or world-model-based approaches.Source: §2.2 · p. 3
  • OGPO extracts goal-reaching and failure-aware supervision from offline robot data and combines offline goal-conditioned RL with SFT regularization.The paper states that this enables improved goal-directed recoverability without additional environment interaction.Source: §2.2 · p. 3

3 Method

OGPO turns offline trajectories into key-state goals and process rewards, then optimizes a VLA policy offline with advantage weighting and SFT regularization.

3.1 Goal-Relabeled Trajectory Segmentation

  • Gripper-state changes divide demonstrations into task-progress segments, which receive semantic local goals and binary completion labels for achieved or unfulfilled key states.Successful segments are labeled complete; an offline VLM annotator evaluates segment completion in failed trajectories.Source: §3.1 · p. 4

3.2 Designing Goal-Conditioned Process Rewards for Value Learning

  • Goal-conditioned process rewards turn segment completion and progress into dense signals while keeping relabeled goals out of the policy and value-function inputs.Rewards combine a step penalty, positive progress credit within achieved segments, and a terminal success bonus or failure penalty.Source: §3.2 · p. 5
  • A value function learns expected discounted returns from the reward-augmented dataset using an n-step Bellman target and a lightweight head on the VLA backbone.States followed by achieved key-state progress receive higher values, while failed or incomplete segments receive lower values.Source: §3.2 · p. 5

3.3 Offline Goal-conditioned Policy Optimization

  • AWAC-style advantage-weighted regression favors actions with higher estimated returns, while SFT regularizes the policy toward feasible demonstrated actions.The two objectives provide complementary signals: offline RL can favor progress beyond demonstration-specific paths, and SFT preserves the expert action prior.Source: §3.3 · pp. 5–6

4 Experiments

Experiments evaluate OGPO against SFT and other baselines on manipulation benchmarks, distribution shifts, and mid-execution perturbations. OGPO improves benchmark performance and robustness, while ablations identify contributions from its reward design and policy objectives.

4.1 Experimental Setup

  • Experiments use LIBERO and MetaWorld, add LIBERO-Plus shifts in viewpoint, initial state, visual noise, and object layout, and test LIBERO-ReAct execution disturbances.Baselines include π0 and π0.5 SFT variants, representative VLA models, and PPO-based online RL.Source: §4.1 · p. 6

4.2 Main Results

  • On LIBERO, OGPO raises average success from 77.1% to 97.1%, exceeding π0.5 full-SFT at 96.85%; on MetaWorld-MT50, π0 improves from 50.8% to 81.0%.The LIBERO comparison starts from the π0.5 few-shot policy; MetaWorld gains are reported across backbones.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • Against full-shot SFT, offline-fine-tuned π0.5 raises overall robustness from 76.7% to 81.8%, with gripper-state perturbation success increasing from 76.8% to 87.7%.The evaluation includes LIBERO-Plus distribution shifts and LIBERO-ReAct mid-episode disturbances, without perturbation-specific fine-tuning.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • With π0 trained for 3,000 steps, OGPO reaches 93.6% on LIBERO-Spatial and 92.3% on LIBERO-Goal, beating SFT by 13.6 and 11.1 points, respectively.Figure 3 compares training efficiency between OGPO and SFT using the π0 backbone.Source: §4.2 · p. 8, Figure 3 · p. 9

4.3 Ablation Study

  • The proposed reward design performs best on LIBERO-Goal and LIBERO-Spatial; removing Rgoal causes the largest drop, underscoring the role of subgoal-completion signals in offline value learning.Without trajectory segmentation and subgoal relabeling, the value function cannot effectively distinguish achieved progress from failed segments.Source: §4.3 · p. 9, Figure 4 · p. 9
  • The full policy objective performs best on both tasks: removing offline RL degrades performance, while removing SFT loses feasible-behavior regularization and permits drift from the pretrained action prior.Figure 4 reports these ablations on the π0 backbone trained for 3,000 steps.Source: §4.3 · p. 9, Figure 4 · p. 9

5 Conclusion and Limitations

OGPO turns offline trajectories into goal-conditioned segments and rewards for recoverable policy learning, while its current segmentation and offline-data dependence constrain where recovery may generalize.

5 Conclusion and Limitations

  • OGPO shifts post-training from action imitation toward key-state reachability using relabeled goals, process rewards, value learning, and advantage-weighted policy optimization with SFT.Relabeled goals shape training rewards but are not additional policy or value-function inputs.Source: §5 · p. 9, A Pseudo-code of OGPO · p. 14
  • The current gripper-state segmentation may not suit tasks without naturally corresponding events, and offline-data quality may limit recovery from severe out-of-distribution deviations.The authors propose more general goal relabeling and greater offline-data diversity as future directions.Source: §5 · p. 10

A Pseudo-code of OGPO

  • OGPO’s training procedure segments demonstration and failed trajectories, assigns goals and completion labels, annotates process rewards, learns values, and optimizes the policy with reward-weighted and SFT losses.The policy is updated using minibatches from the relabeled dataset and advantages computed from the learned value function.Source: A Pseudo-code of OGPO · p. 14

B.1 Gripper-Based Trajectory Segmentation

B.2 Goal Relabel

  • Each segment receives a semantic local goal based on the instruction, gripper transition, and visual state change, such as grasping, transporting, placing, or manipulating an articulated object.The relabeled trajectory becomes a sequence of goal-conditioned segments.Source: §B.2 · p. 15

B.3 Completion Labeling

OGPO labels completion at the segment level, preserving successful local progress even in failed trajectories through offline VLM annotation.

B.3 Completion Labeling

  • Each segment receives a binary label indicating whether it completes its assigned goal; valid segments in successful trajectories are labeled successful.The segment-end observation represents the achieved task-progress key state.Source: §B.3 · p. 15
  • For failed trajectories, a VLM judges whether each segment’s ending visual state satisfies its assigned goal, rather than labeling every segment from the task-level failure.This localizes the failure stage and supplies completion supervision for value learning.Source: §B.3 · p. 15

B.4 VLM Annotation Prompt for Failed Segments

  • The failed-segment annotation prompt supplies the task instruction, candidate goal, and representative observations, and requests a completion label with a short explanation.The annotator is Qwen2.5-VL-72B-Instruct.Source: §B.4 · p. 16, Figure 5 · p. 16

B.5 Annotated Cases

The annotated cases show how segment-level labels distinguish achieved intermediate goals from a later failure in an unsuccessful trajectory.

B.5 Annotated Cases

  • In the successful example, gripper-based segmentation yields goal segments that are all labeled achieved.The displayed frames are sampled every 10 time steps for illustration.Source: §B.5 · p. 16
  • In the failed example, early segments complete meaningful goals even though the full trajectory fails.Source: §B.5 · p. 16
  • The annotation preserves those achieved segments and marks the later segment that misses its assigned goal as negative experience.Source: §B.5 · p. 16

C Reward Design and Value Analysis

OGPO uses segment completion and progress to provide denser reward signals, and its learned values rise around semantically meaningful task-progress states.

C Reward Design and Value Analysis

  • Completed segments receive dense rewards based on normalized within-segment progress, while failed segments receive no progressive positive reward.The reward combines a step penalty, progress reward, and segment-end success or failure term.Source: C Reward Design and Value Analysis · p. 16
  • The step penalty discourages long or redundant action sequences, progress rewards credit movement toward a completed goal, and terminal terms reward success or penalize failure.Source: C Reward Design and Value Analysis · p. 17
  • Learned value curves show local high-value regions aligned with milestones such as approaching or grasping an object, opening a drawer, and placing the object.This pattern suggests the value function captures intermediate progress as well as final task success.Source: C Reward Design and Value Analysis · p. 17

D Hyperparameter Analysis

The objective-weight analysis describes three complementary terms, and reports that using all three yields more stable and effective policy improvement than relying on any single term. Learned value peaks also align with semantic task-progress states.

D Hyperparameter Analysis

  • The supervised fine-tuning term anchors imitation, while offline RL distinguishes actions by their estimated long-term contribution to goal completion.Source: D Hyperparameter Analysis · p. 17
  • Local high-value frames align with semantic task-progress states, suggesting the learned value provides dense progress signals beyond final task success.Examples include placing a bowl on a plate, opening a drawer, and moving a plate toward the stove.Source: Figure 8 · p. 18
  • Using all three objective components yields more stable and effective policy improvement than relying on any single term alone.The components are supervised fine-tuning on achieved goal segments, offline RL, and negative suppression on failed goal segments.Source: D Hyperparameter Analysis · pp. 17–18

E Implementation Details

The offline fine-tuning implementation matches its corresponding SFT baseline on preprocessing, action normalization, backbone initialization, and optimization setup. The value model is trained before policy fine-tuning and used only to compute advantage weights during offline training.

E Implementation Details

  • For fair comparison, the implementation matches the corresponding SFT baseline's preprocessing, action normalization, backbone initialization, and optimization setup.Source: E Implementation Details · p. 18
  • The value model is trained before policy fine-tuning and used only during offline training to compute advantage weights.Source: E Implementation Details · p. 18
  • The implementation tables list value-model training settings and offline-objective hyperparameters.Source: Table 6 · p. 19, Table 7 · p. 19

F Computational Cost Analysis

The method performs reward annotation, value learning, and policy optimization on a fixed offline dataset without collecting policy rollouts. Its reported computational-cost comparison is substantially more efficient than online RL-based training, though the table compares different numbers of LIBERO suites.

F Computational Cost Analysis

  • The computational-cost comparison reports the offline method as substantially more efficient than online RL-based training, while GRPO covers one LIBERO suite and the offline method covers all four.Source: F Computational Cost Analysis · p. 19, Table 8 · p. 19
  • Reward annotation, value learning, and policy optimization use a fixed offline dataset, with no policy rollout collection during training.Source: F Computational Cost Analysis · p. 19
  • Avoiding online environment interaction is described as making scaling across task suites easier and robotic manipulation safer when rollouts are expensive or risky.Source: F Computational Cost Analysis · p. 19

Related papers

Loading 2610.12914v1…