PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation

Zhide Zhong, Haodong Yan, Junfeng Li, Junjie He, Tianran Zhang, Haoang Li

arXiv:2603.26666v1cs.RO

TL;DR

SFT and online RL each have limitations for VLA post-training: SFT faces distribution shift, while RL relies on sparse rewards. VLA-OPD distills a teacher’s dense guidance on student-generated trajectories using Reverse-KL, and reports improved robustness over SFT and sample efficiency over RL across LIBERO and RoboTwin2.0.

  • Problem

    Offline SFT is vulnerable to distribution shift and catastrophic forgetting, whereas online RL relies on sparse rewards and is sample-inefficient; Forward-KL and Hard-CE also risk entropy explosion and collapse, respectively.

    Source: §1 · p. 2

  • Method

    VLA-OPD uses a frozen expert teacher to provide dense, token-level supervision on student-generated trajectories and trains the student with a Reverse-KL objective.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Across LIBERO and RoboTwin2.0, VLA-OPD reports greater robustness than SFT and fewer training steps than on-policy RL baselines, while mitigating catastrophic forgetting.

    Source: §1 · p. 3

  • Takeaways & Limitations

    VLA-OPD offers a sample-efficient way to transfer robust behaviors from existing teachers into new, upgraded, or unified generalist students.

    Source: §1 · p. 2

Abstract

from arXiv · show

Although pre-trained Vision-Language-Action (VLA) models exhibit impressive generalization in robotic manipulation, post-training remains crucial to ensure reliable performance during deployment. However, standard offline Supervised Fine-Tuning (SFT) suffers from distribution shifts and catastrophic forgetting of pre-trained capabilities, while online Reinforcement Learning (RL) struggles with sparse rewards and poor sample efficiency. In this paper, we propose On-Policy VLA Distillation (VLA-OPD), a framework bridging the efficiency of SFT with the robustness of RL. Instead of relying on sparse environmental rewards, VLA-OPD leverages an expert teacher to provide dense, token-level supervision on the student's self-generated trajectories. This enables active error correction on policy-induced states while preserving pre-trained general capabilities through gentle alignment. Crucially, we formulate VLA-OPD via a Reverse-KL objective. Unlike standard Forward-KL that induces mode-covering entropy explosion, or Hard-CE that causes premature entropy collapse, our bounded mode-seeking objective ensures stable policy learning by filtering out the teacher's epistemic uncertainty while maintaining action diversity. Experiments on LIBERO and RoboTwin2.0 benchmarks demonstrate that VLA-OPD significantly improves sample efficiency over RL and robustness over SFT, while effectively mitigating catastrophic forgetting during post-training.

1. Introduction

Pre-trained VLAs generalize broadly, but adapting them for reliable downstream execution requires post-training. VLA-OPD combines on-policy student trajectories, dense teacher supervision, and Reverse-KL alignment to address limitations of SFT and online RL while preserving general capabilities.

1. Introduction

  • SFT trains on static expert demonstrations, whereas online RL exposes policies to their own induced states but depends on sparse rewards.SFT also requires large-scale, high-quality demonstrations to cover diverse scenes.Source: §1 · pp. 1–2
  • Reverse-KL distillation uses bounded mode-seeking to avoid Forward-KL entropy explosion and Hard-CE entropy collapse while retaining action diversity.The paper describes this alignment as supporting stable updates and preserving pre-trained generalist capabilities.Source: §1 · pp. 2–3
  • VLA-OPD separates costly RL exploration from student optimization and transfers robust behaviors from existing expert teachers into student backbones.The paper describes this as a sample-efficient path for developing new, upgraded, or unified generalist students.Source: §1 · p. 2
  • VLA-OPD provides dense, token-level teacher supervision on student-generated trajectories, combining SFT’s convergence efficiency with RL’s robustness.The framework aims to mitigate SFT exposure bias and sparse-reward RL sample inefficiency.Source: §1 · p. 2, Table 1 · p. 2
  • VLA-OPD improves robustness over SFT and requires fewer training steps than on-policy RL baselines across LIBERO and RoboTwin2.0.The reported results also indicate mitigation of catastrophic forgetting during post-training.Source: §1 · p. 3

2. Preliminaries

The preliminaries frame robotic manipulation as policy learning for task success, then contrast static demonstration-based SFT with online RL. VLA-OPD’s overview uses student rollouts, teacher labels, and Reverse-KL optimization to address the sparse feedback of RL.

2.1. Problem Formulation

  • Robotic manipulation is formulated as an MDP in which a VLA policy maps visual observations and language instructions to actions to maximize task success.The formulation specifies state, action, transition, reward, and discount components.Source: §2.1 · p. 3

2.2. Supervised Fine-Tuning (SFT)

  • SFT maximizes expert-action likelihood on static demonstrations, but trains on expert states while being evaluated on student-induced states.This off-policy mismatch is identified as the source of distribution shift.Source: §2.2 · p. 3

2.3. Online RL with Sparse Outcome Rewards

  • VLA-OPD samples student trajectories, obtains teacher action labels for visited states, and optimizes the student against the teacher distribution with Reverse-KL.The figure contrasts Reverse-KL’s mode-seeking behavior with Forward-KL’s mass-covering entropy explosion.Source: Figure 1 · p. 4, §2.3 · p. 4
  • GRPO samples trajectory groups and optimizes outcome rewards, but robotic tasks often provide only binary completion feedback, causing high-variance optimization and poor sample efficiency.GRPO avoids a separate value network through group-based relative advantage normalization.Source: §2.3 · pp. 3–4

3. Methodology

VLA-OPD aligns a student VLA to a robust teacher through dense supervision on the student’s own trajectories. Its on-policy updates address distribution shift and forgetting, while Reverse-KL provides bounded mode-seeking optimization.

3.1. Framework Overview

  • The framework uses a frozen expert teacher to guide a student that samples and learns from its own environment trajectories.The student’s three-phase cycle samples trajectories, receives teacher action logits at visited states, and updates with token-level Reverse-KL reward.Source: §3.1 · pp. 4–5

3.2. On-Policy Trajectory Sampling

  • On-policy sampling exposes student-induced failure states for training, addressing the mismatch between restricted expert states and states encountered during evaluation.The SFT-initialized student may accumulate execution errors and enter unfamiliar states.Source: §3.2 · p. 6
  • Anchoring updates to the student’s current behavioral manifold is presented as a way to mitigate catastrophic forgetting and preserve pre-trained capabilities.The paper contrasts this with offline SFT updates on fixed, disjoint targets.Source: §3.2 · p. 6

3.3. Dense Teacher Supervision

  • Dense token-level teacher labels replace sparse outcome rewards and provide recovery guidance on student-visited, including out-of-distribution, states.The paper says this converts delayed RL feedback into immediate supervision and can accelerate convergence.Source: §3.3 · p. 6

3.4. Optimization Objective and Analysis

  • Reverse-KL is mode-seeking: it filters teacher-tail uncertainty while retaining stochasticity within valid modes, unlike Forward-KL’s entropy explosion and Hard-CE’s entropy collapse.The reported ablation says Reverse-KL maintains bounded entropy and achieves the highest, most stable task success rate; policy gradients are group-averaged and use raw Reverse-KL reward as the advantage.Source: §3.4 · pp. 6–7

4. Experiments

Experiments evaluate VLA-OPD’s efficiency and efficacy across LIBERO and RoboTwin2.0, its retention of unseen capabilities, and two ablation choices. Results favor distillation for sample efficiency, policy performance, and preserving capabilities, with Reverse KL and smaller sampling groups also performing well.

4.1. Experimental Setup

  • The evaluation covers four LIBERO suites initialized from one demonstration per task and four dual-arm RoboTwin2.0 tasks initialized from 1,000 demonstrations per task.The baselines include sparse-reward GRPO and offline SFT on full expert datasets.Source: §4.1 · p. 8, §4.2 · p. 8

4.2. Main Results: Efficiency and Efficacy

  • On LIBERO-Long, near 80% success arrives in 50 steps versus GRPO’s over 150, a 3× speedup; Distill + GRPO surpasses GRPO’s final convergence.On LIBERO-Object, distillation exceeds 90% success within 10 steps.Source: §4.2 · p. 8, Figure 2 · p. 9
  • On LIBERO, average success rises from 48.9% (OpenVLA-OFT) to 87.4% (Distill) and 93.4% (Distill + GRPO), near the teacher’s 93.9%.VLA-OPD’s 1-traj performance is comparable to full-dataset methods.Source: Table 2 · p. 9, §4.2 · p. 9
  • On RoboTwin2.0, Distill raises average success from 45.2% to 71.1%, nearly matching the teacher’s 74.0% across complex dual-arm tasks.Students started from 1,000 demonstrations per task via single-task SFT.Source: Table 3 · p. 10, §4.2 · p. 10

4.3. Mitigating Catastrophic Forgetting via On-Policy Alignment

  • Offline SFT’s unseen-task performance collapses as seen-task success rises, while on-policy RL and VLA-OPD largely preserve unseen capabilities.VLA-OPD matches or exceeds RL on several held-out tasks.Source: §4.3 · p. 10, Figure 3 · p. 11

4.4. Ablation Studies

  • On RoboTwin2.0 Beat Block Hammer, Reverse KL improves steadily while Forward KL drops over 50% early and Hard CE plateaus; bounded mode-seeking avoids entropy extremes.Forward KL’s mode-covering produces entropy explosion, whereas Hard CE’s argmax matching causes premature entropy collapse and reduces action diversity.Source: §4.4 · p. 11, Figure 4 · p. 12
  • With group sampling, G = 8 yields the smoothest optimization and highest final success rate (∼89%), while G = 2 steadily climbs to over 80%, demonstrating competitive performance and reducing rollout and teacher-inference overhead.Source: §4.4 · p. 12

5. Related Work

The section contrasts offline SFT’s distribution-shift and alignment-objective problems with online RL’s sparse-reward inefficiency. VLA-OPD combines on-policy exploration with dense teacher guidance and Reverse-KL alignment.

5.1. Offline SFT for Vision-Language-Action Models

  • Offline SFT can suffer covariate shift and compounding errors, while Hard-CE may collapse entropy and Forward-KL may cause entropy explosion in uncertain OOD states.SFT’s dense supervision and fast convergence come with a reliance on high-quality demonstrations and risks from static data [O’Neill et al. (2024), Black et al. (2024); Zhu et al. (2025), Lai et al. (2025), Chu et al. (2025), Shenfeld et al. (2025)].Source: §5.1 · p. 13

5.2. Online RL for Vision-Language-Action Models

  • To address offline SFT distribution shift, online RL aligns models with policy-induced state distributions and enables recovery behaviors from execution trajectories; standard RL for billion-parameter VLAs faces sparse rewards, low sample efficiency, and high-variance optimization.Source: §5.2 · p. 13
  • VLA-OPD uses on-policy RL with dense, token-level teacher guidance and Reverse-KL to address the limitations of imitation learning and sparse-reward RL.It encourages commitment to the teacher’s most confident action mode in uncertain OOD states while avoiding the entropy extremes of Forward-KL and Hard-CE.Source: §5.2 · p. 13

6. Conclusion

VLA-OPD combines offline SFT’s sample efficiency with online RL’s closed-loop robustness through on-policy exploration and dense teacher guidance. The conclusion reports stable Reverse-KL updates, superior success rates without catastrophic forgetting, and identifies reliance on specific teachers as future work.

6. Conclusion

  • VLA-OPD bridges offline SFT’s sample efficiency and online RL’s closed-loop robustness using active exploration and dense, token-level teacher guidance.The framework is presented as mitigating compounding errors and bypassing sparse-reward RL’s extreme sample inefficiency.Source: §6 · p. 13
  • Reverse-KL avoids Forward-KL and Hard-CE entropy extremes in OOD states and is reported to ensure stable, robust policy updates.The conclusion identifies this objective as a critical insight of the framework.Source: §6 · p. 13
  • The conclusion reports superior success rates on complex robotic manipulation tasks without catastrophic forgetting and names dependence on specific teacher models as future work.Source: §6 · p. 13

Related papers

Loading 2603.26666v1…