PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models

Liangzhi Shi, Shuaihang Chen, Feng Gao, Yinuo Chen, Kang Chen, Tonghe Zhang, Hongzhi Zang, Jiakai Zhou, Weinan Zhang, Chao Yu, Yu Wang

arXiv:2602.12628v4cs.RO

TL;DR

VLA sim-real co-training often treats simulation as static demonstrations, limiting its use of scalable closed-loop interaction. RL-Co warm-starts with mixed real and simulated demonstrations, then applies simulation RL anchored by real-world supervision, improving real-world success, robustness, and data efficiency while remaining scoped to tabletop manipulation with one robot embodiment.

  • Problem

    Most sim-real co-training methods treat simulation as static demonstrations rather than exploiting large-scale closed-loop interaction.

    Source: §1 · p. 2, §3.2 · p. 4

  • Method

    RL-Co uses supervised co-training on real and simulated demonstrations for initialization, followed by simulation reinforcement learning with auxiliary real-world SFT.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    RL-Co outperforms real-only fine-tuning and SFT-based co-training across tasks and VLA models, improving real-world success, robustness to distribution shifts, and data efficiency.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Interactive simulation provides benefits that static imitation objectives alone cannot fully realize for real-robot VLA deployment.

    Source: §6 · p. 9

    The study focuses on tabletop manipulation with a single robot embodiment and does not explore heterogeneous sim-real settings.

    Source: §7 · p. 9

Abstract

from arXiv · show

Simulation offers a scalable and low-cost way to enrich vision-language-action (VLA) training, reducing reliance on expensive real-robot demonstrations. However, most sim-real co-training methods rely on supervised fine-tuning (SFT), which treats simulation as a static source of demonstrations and does not exploit large-scale closed-loop interaction. Consequently, real-world gains and generalization are often limited. In this paper, we propose an RL-based sim-real Co-training (RL-Co) framework that leverages interactive simulation while preserving real-world capabilities. Our method follows a generic two-stage design: we first warm-start the policy with SFT on a mixture of real and simulated demonstrations, then fine-tune it with reinforcement learning in simulation while adding an auxiliary supervised loss on real-world data to anchor the policy and mitigate catastrophic forgetting. We evaluate our framework on four real-world tabletop manipulation tasks using two representative VLA architectures, OpenVLA and $π_{0.5}$, and observe consistent improvements over real-only fine-tuning and SFT-based co-training, including +24% real-world success on OpenVLA and +20% on $π_{0.5}$. Beyond higher success rates, RL co-training yields stronger generalization to unseen task variations and substantially improved real-world data efficiency, providing a practical and scalable pathway for leveraging simulation to enhance real-robot deployment.

1 Introduction

VLA policies remain sensitive to real-world distribution shifts, while simulation offers scalable interaction but introduces a sim-to-real gap. RL-Co addresses these limitations by combining interactive simulation RL with real-world supervision and improves success and generalization across models and tasks.

  • Limited real demonstrations leave VLA policies brittle under novel scenes, object variations, and closed-loop execution shifts.Source: §1 · p. 1
  • Simulation scales environments, trajectories, and closed-loop interaction, but sim-to-real methods often require careful randomization, reconstruction, or task-specific tuning.Source: §1 · pp. 1–2
  • Sim-real co-training avoids direct transfer by adding real-world data, yet most existing methods treat simulation as static demonstrations and underexplore closed-loop interaction.Source: §1 · p. 2
  • RL-Co initializes VLA policies with supervised sim-real co-training, then applies simulation reinforcement learning with auxiliary real-world supervision as a cross-domain anchor.Source: §1 · p. 2
  • RL-Co improves real-world success over real-only SFT and SFT-based co-training across OpenVLA and π0.5, while improving unseen-variation generalization and reducing real-demonstration needs.Source: §1 · p. 2

2 Related Works

VLA post-training remains sensitive to distribution shifts, motivating reinforcement learning approaches that use simulation to provide interactive improvement without costly direct real-world exploration. However, simulation transfer still faces visual and dynamics discrepancies.

  • VLA deployment remains sensitive to shifts across scenes, objects, and execution states despite large-scale demonstrations and visual-language priors.Source: §2.1 · p. 2
  • Supervised fine-tuning is vulnerable to covariate shift and compounding closed-loop errors, while direct real-world RL is constrained by safety, reset cost, and sample efficiency.Source: §2.2 · p. 3
  • Simulation-based RL can accelerate post-training, but direct transfer remains limited by visual and dynamics discrepancies, motivating domain randomization and reconstruction-based pipelines.Source: §2.3 · p. 3

3 Preliminaries

The paper models real and simulated manipulation as related partially observable decision processes with shared task interfaces but possible visual and dynamics discrepancies. RL-Co combines demonstration-based initialization and real-domain anchoring with reward-guided simulation interaction.

  • The real and simulation task domains share embodiment, action interface, language instruction, and task-level initial-state distribution while allowing visual and dynamics discrepancies.Source: §3.1 · p. 3
  • RL-Co combines SFT for initialization and real-domain anchoring with RL for closed-loop improvement in simulation.Source: §3.1 · p. 3
  • SFT-based sim-real co-training jointly fine-tunes a VLA policy on real-world and simulated demonstrations.Source: §3.2 · p. 3
  • RL-Co uses a two-stage design: supervised co-training initializes the policy, then simulation RL improves it while an auxiliary supervised objective preserves real-world capabilities.Source: Figure 2 · p. 4
  • SFT-based co-training cannot explicitly use reward feedback or online interaction because it remains an imitation objective.Source: §3.2 · p. 4

4 Method

RL-Co first equips a VLA policy with real and simulated demonstrations, then expands competence through simulation interaction while preserving real-world performance with supervised regularization. The framework uses real-world SFT to anchor simulation-driven updates to deployable behavior.

Method overview

  • RL-Co initializes the policy through supervised co-training on real-world and simulated demonstrations before applying simulation reinforcement learning with real-world supervision.Source: §4 · p. 4

Stage I

  • Stage I uses mixed demonstrations to absorb task-specific real-world knowledge and develop sufficient simulation competence for nontrivial reinforcement learning.Source: §4.1 · p. 4

Stage II: Sim-Real Co-Training with Real-Regularized RL

  • Stage II uses online simulation interaction to expand competence while an auxiliary real-world SFT objective prevents degradation of real-world performance.Source: §4.2 · p. 4
  • β balances reinforcement learning updates and preservation of real-world knowledge.Source: §4.2 · p. 5
  • The real-world supervision term anchors updates to demonstrations and mitigates catastrophic forgetting while the RL term explores diverse behaviors and maximizes task rewards.Source: §4.2 · p. 5

5 Experiments

The experiments evaluate RL-Co across four tabletop tasks and two VLA families, comparing it with real-only SFT and SFT-based sim-real co-training. Results assess deployment success, unseen-setting robustness, visual-diversity explanations, two-stage mechanisms, and real-data efficiency.

Experimental Setting

  • The evaluation spans four tabletop manipulation tasks, two VLA families, and real-world success-rate comparisons against real-only SFT and SFT-based sim-real co-training.The models are OpenVLA and π0.5; results are reported as success rate (SR, %) with mean ± standard deviation.Source: §5.1 · pp. 5–6, §5.2.1 · p. 6, Table 1 · p. 6

Main Comparison Across Tasks and VLA Models

  • RL-Co achieves the strongest overall real-world success, outperforming both supervised baselines across the evaluated tasks and model families.The comparison uses limited real demonstrations and imperfect simulation.Source: §5.2.1 · p. 6

Generalization Under Distribution Shifts

  • RL-Co maintains the strongest unseen performance and smallest drops under novel objects and perturbed initial poses, while real-only SFT degrades sharply.The distribution-shift evaluation uses Pick and Place with π0.5, where methods are competitive in-distribution.Source: §5.2.2 · p. 6, Figure 4 · p. 6

Targeted Comparison with Stronger Visual-Diversity Baselines

  • Visual augmentation alone is insufficient: heavy domain randomization and video-based augmentation remain below RL-Co, which expands action-level behavior through simulation interaction.The targeted comparison evaluates Pick and Place with π0.5 under the same real-world protocol.Source: §5.2.3 · p. 7, Table 2 · p. 7

Effect of Simulation SFT Initialization

  • Simulation SFT initialization makes RL optimization efficient, whereas omitting it leaves the policy near-trivial after more than three million interaction steps.The ablation compares real-only SFT initialization with full sim-real SFT initialization.Source: §5.3.1 · p. 7, Figure 5 · p. 7

Two-Stage Design and Real-Data Efficiency

  • Real-world supervision in both stages is necessary for deployment, and RL co-training improves over SFT co-training across tested mixture ratios and regularization weights.Removing Stage II anchoring causes domain drift, while removing supervision from both stages produces collapse; RL-Co also improves real-data efficiency.Source: §5.3.2 · p. 8, §5.3.3 · p. 8, §5.4 · p. 8

6 Conclusion

RL-Co combines real and simulated demonstrations with reinforcement learning in simulation and real-world supervision, improving performance beyond supervised co-training alone. Experiments show gains in real-world success, robustness to distribution shifts, and data efficiency.

  • RL-Co initializes with real and simulated demonstrations, then uses reinforcement learning with real-world supervision to preserve deployable behaviors.Source: §6 · p. 8
  • RL-Co outperforms real-only fine-tuning and SFT-based co-training across tasks and VLA models.Source: §6 · p. 8
  • RL-Co improves real-world success, robustness to distribution shifts, and data efficiency.Source: §6 · p. 8
  • Interactive simulation provides benefits that static imitation objectives alone cannot fully realize.Source: §6 · p. 9

7 Limitations

The study is limited to tabletop manipulation with a single robot embodiment and does not evaluate heterogeneous sim-real settings. Although RL-Co improves real-world success, performance remains below 100%.

  • The evaluation covers tabletop manipulation with one robot embodiment and excludes heterogeneous sim-real settings.Source: §7 · p. 9
  • RL-Co improves real-world success, but its performance remains below 100%.Source: §7 · p. 9

A Detailed Problem Setup and Post-Training Objectives

The paper formulates real and simulated manipulation as related partially observable decision processes and defines a VLA policy that maps recent observations and language to future actions. It then describes supervised imitation and reinforcement-learning objectives, with RL-Co adding real-world supervision during simulation-based policy updates.

A.1 Problem Formulation

  • Real and simulated tasks are modeled as POMDPs with domain-specific states, observations, transition dynamics, shared actions, language instructions, rewards, initial states, and discounting.Source: §A.1 · p. 15
  • The VLA policy conditions on recent observations and a language instruction to predict future actions over a fixed horizon.Source: §A.1 · p. 15

A.2 Post-Training Objectives

  • Post-training adapts a pretrained VLA policy using either expert demonstrations or online environmental interaction.Source: §A.2 · p. 15

A.2.1 Supervised Fine-Tuning (SFT)

  • Supervised fine-tuning minimizes the discrepancy between predicted and expert action sequences, with the loss depending on the VLA architecture and action representation.Source: §A.2.1 · p. 16

A.2.2 Reinforcement Learning (RL) Fine-Tuning

  • Reinforcement learning fine-tuning maximizes expected discounted return from policy-sampled actions and environment transitions.Source: §A.2.2 · p. 16
  • RL-Co adds a real-world supervised fine-tuning objective during reinforcement-learning updates in simulation.Source: §A.2.2 · p. 16

B Dataset Generation Details

The study combines teleoperated real demonstrations with large-scale simulated trajectories, evaluates four tabletop tasks using aligned robot and camera interfaces, and tests imperfect simulation anchored by real data.

Real-world data

  • 20–50 successful teleoperated trajectories per task form the real-world dataset Dreal.Experts use a 3D Space-Mouse, and actions are recorded as end-effector delta-control commands.Source: §B.1 · p. 16

Simulation data

  • 1,000 successful simulated trajectories per task form Dsim after MimicGen replays real expert trajectories and prunes free-space motion.The simulation data are generated in ManiSkill from real-world seed behaviors, retaining task-relevant key stages.Source: §B.2 · p. 16

Tasks and setup

  • Four tasks require perception, language grounding, and control: Pick and Place, Push Cube via Instruction, Open Drawer, and Close Drawer.The environments use a Franka Panda with seven-degree-of-freedom end-effector delta control and fixed RGB observations.Source: C Real-world Environment Setup · p. 17, §D.1 · p. 17

Generalization design

  • Real-world evaluation separates demonstrated and unseen object variations, including excluded cube colors and irregular-shaped objects absent from demonstrations.These splits support generalization evaluation beyond the demonstrated object set.Source: §D.2 · pp. 17–18, Figure 9 · p. 18

Simulation fidelity

  • The simulator need not be photorealistic or a fully accurate digital twin because real-data supervision anchors the policy across visual differences.Object identities, materials, textures, lighting, and backgrounds are not precisely matched between domains.Source: §E.1 · p. 20

Reward design

  • Task rewards combine dense shaping and terminal success signals, with reward design varying by task and model.Pick and Place uses dense rewards for OpenVLA and sparse rewards for π0.5; drawer tasks issue terminal rewards of 5.0.Source: §E.2 · pp. 21–22

F Additional Dexterous Evaluation

The framework is also tested on Peg Insertion, a more dexterous task requiring precise contact and alignment, where RL-Co improves real-world success over SFT-based co-training.

Task

  • Peg Insertion requires aligning a grasped 2 × 2 × 12 cm peg with a 2.25 × 2.25 cm square hole before insertion.The peg starts 1–5 cm above a horizontally perturbed hole, making the task more precision-demanding than the main tabletop experiments.Source: F Additional Dexterous Evaluation · p. 22, Figure 12 · p. 23

Result

G Sim–Real Correlation During RL Co-Training

During RL co-training, simulation and real-world success generally improve together, although their relationship is imperfect because sim–real alignment is incomplete.

Correlation

Practical implication

  • Simulation convergence is used as a practical checkpoint-selection signal, while real demonstrations reduce domain drift during policy improvement.The positive correlation is useful but not perfect under imperfect sim–real alignment.Source: G Sim–Real Correlation During RL… · p. 23

H.1 One-Stage RL Co-Training

Removing Stage I task-specific SFT prevents effective task acquisition, while static simulation-expert distillation helps but remains below direct RL-Co policy improvement.

One-stage RL co-training

  • The one-stage variant achieves 0.0% real-world success and near-zero simulation success after several million interaction steps.It removes Stage I task-specific SFT and applies RL co-training directly to the pretrained VLA policy.Source: §H.1 · p. 24, Figure 14 · p. 24

Interpretation

  • Stage I SFT initialization provides the task-level behavior prior required for subsequent RL optimization.The result supports a two-stage design rather than replacing warm-starting with a single RL phase.Source: §H.1 · p. 24

Distillation comparison

  • The RialTo-style distillation baseline reaches 75.0% real-world success but remains below RL-Co.A simulation expert generates 2,000 trajectories that are then used for SFT co-training with 50 real demonstrations.Source: §H.2 · p. 24

Related papers

Loading 2602.12628v4…