Source-linked AI summary
Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models
Liangzhi Shi, Shuaihang Chen, Feng Gao, Yinuo Chen, Kang Chen, Tonghe Zhang, Hongzhi Zang, Jiakai Zhou, Weinan Zhang, Chao Yu, Yu Wang
TL;DR
VLA sim-real co-training often treats simulation as static demonstrations, limiting its use of scalable closed-loop interaction. RL-Co warm-starts with mixed real and simulated demonstrations, then applies simulation RL anchored by real-world supervision, improving real-world success, robustness, and data efficiency while remaining scoped to tabletop manipulation with one robot embodiment.
Problem
Most sim-real co-training methods treat simulation as static demonstrations rather than exploiting large-scale closed-loop interaction.
Method
RL-Co uses supervised co-training on real and simulated demonstrations for initialization, followed by simulation reinforcement learning with auxiliary real-world SFT.
Results
RL-Co outperforms real-only fine-tuning and SFT-based co-training across tasks and VLA models, improving real-world success, robustness to distribution shifts, and data efficiency.
Takeaways & Limitations
Interactive simulation provides benefits that static imitation objectives alone cannot fully realize for real-robot VLA deployment.
Takeaways & Limitations
The study focuses on tabletop manipulation with a single robot embodiment and does not explore heterogeneous sim-real settings.
Abstract
from arXiv · showhide
Simulation offers a scalable and low-cost way to enrich vision-language-action (VLA) training, reducing reliance on expensive real-robot demonstrations. However, most sim-real co-training methods rely on supervised fine-tuning (SFT), which treats simulation as a static source of demonstrations and does not exploit large-scale closed-loop interaction. Consequently, real-world gains and generalization are often limited. In this paper, we propose an RL-based sim-real Co-training (RL-Co) framework that leverages interactive simulation while preserving real-world capabilities. Our method follows a generic two-stage design: we first warm-start the policy with SFT on a mixture of real and simulated demonstrations, then fine-tune it with reinforcement learning in simulation while adding an auxiliary supervised loss on real-world data to anchor the policy and mitigate catastrophic forgetting. We evaluate our framework on four real-world tabletop manipulation tasks using two representative VLA architectures, OpenVLA and $π_{0.5}$, and observe consistent improvements over real-only fine-tuning and SFT-based co-training, including +24% real-world success on OpenVLA and +20% on $π_{0.5}$. Beyond higher success rates, RL co-training yields stronger generalization to unseen task variations and substantially improved real-world data efficiency, providing a practical and scalable pathway for leveraging simulation to enhance real-robot deployment.
1 Introduction
VLA policies remain sensitive to real-world distribution shifts, while simulation offers scalable interaction but introduces a sim-to-real gap. RL-Co addresses these limitations by combining interactive simulation RL with real-world supervision and improves success and generalization across models and tasks.
- Limited real demonstrations leave VLA policies brittle under novel scenes, object variations, and closed-loop execution shifts.
- Simulation scales environments, trajectories, and closed-loop interaction, but sim-to-real methods often require careful randomization, reconstruction, or task-specific tuning.
- Sim-real co-training avoids direct transfer by adding real-world data, yet most existing methods treat simulation as static demonstrations and underexplore closed-loop interaction.
- RL-Co initializes VLA policies with supervised sim-real co-training, then applies simulation reinforcement learning with auxiliary real-world supervision as a cross-domain anchor.
- RL-Co improves real-world success over real-only SFT and SFT-based co-training across OpenVLA and π0.5, while improving unseen-variation generalization and reducing real-demonstration needs.
2 Related Works
VLA post-training remains sensitive to distribution shifts, motivating reinforcement learning approaches that use simulation to provide interactive improvement without costly direct real-world exploration. However, simulation transfer still faces visual and dynamics discrepancies.
- VLA deployment remains sensitive to shifts across scenes, objects, and execution states despite large-scale demonstrations and visual-language priors.
- Supervised fine-tuning is vulnerable to covariate shift and compounding closed-loop errors, while direct real-world RL is constrained by safety, reset cost, and sample efficiency.
- Simulation-based RL can accelerate post-training, but direct transfer remains limited by visual and dynamics discrepancies, motivating domain randomization and reconstruction-based pipelines.
3 Preliminaries
The paper models real and simulated manipulation as related partially observable decision processes with shared task interfaces but possible visual and dynamics discrepancies. RL-Co combines demonstration-based initialization and real-domain anchoring with reward-guided simulation interaction.
- The real and simulation task domains share embodiment, action interface, language instruction, and task-level initial-state distribution while allowing visual and dynamics discrepancies.
- RL-Co combines SFT for initialization and real-domain anchoring with RL for closed-loop improvement in simulation.
- SFT-based sim-real co-training jointly fine-tunes a VLA policy on real-world and simulated demonstrations.
- RL-Co uses a two-stage design: supervised co-training initializes the policy, then simulation RL improves it while an auxiliary supervised objective preserves real-world capabilities.
- SFT-based co-training cannot explicitly use reward feedback or online interaction because it remains an imitation objective.
4 Method
RL-Co first equips a VLA policy with real and simulated demonstrations, then expands competence through simulation interaction while preserving real-world performance with supervised regularization. The framework uses real-world SFT to anchor simulation-driven updates to deployable behavior.
- Method overview: RL-Co initializes the policy through supervised co-training on real-world and simulated demonstrations before applying simulation reinforcement learning with real-world supervision.
- Stage I: Stage I uses mixed demonstrations to absorb task-specific real-world knowledge and develop sufficient simulation competence for nontrivial reinforcement learning.
- Stage II: Sim-Real Co-Training with Real-Regularized RL: Stage II uses online simulation interaction to expand competence while an auxiliary real-world SFT objective prevents degradation of real-world performance.
- Stage II: Sim-Real Co-Training with Real-Regularized RL: β balances reinforcement learning updates and preservation of real-world knowledge.
- Stage II: Sim-Real Co-Training with Real-Regularized RL: The real-world supervision term anchors updates to demonstrations and mitigates catastrophic forgetting while the RL term explores diverse behaviors and maximizes task rewards.
5 Experiments
The experiments evaluate RL-Co across four tabletop tasks and two VLA families, comparing it with real-only SFT and SFT-based sim-real co-training. Results assess deployment success, unseen-setting robustness, visual-diversity explanations, two-stage mechanisms, and real-data efficiency.
- Experimental Setting: The evaluation spans four tabletop manipulation tasks, two VLA families, and real-world success-rate comparisons against real-only SFT and SFT-based sim-real co-training.The models are OpenVLA and π0.5; results are reported as success rate (SR, %) with mean ± standard deviation.
- Main Comparison Across Tasks and VLA Models: RL-Co achieves the strongest overall real-world success, outperforming both supervised baselines across the evaluated tasks and model families.The comparison uses limited real demonstrations and imperfect simulation.
- Generalization Under Distribution Shifts: RL-Co maintains the strongest unseen performance and smallest drops under novel objects and perturbed initial poses, while real-only SFT degrades sharply.The distribution-shift evaluation uses Pick and Place with π0.5, where methods are competitive in-distribution.
- Targeted Comparison with Stronger Visual-Diversity Baselines: Visual augmentation alone is insufficient: heavy domain randomization and video-based augmentation remain below RL-Co, which expands action-level behavior through simulation interaction.The targeted comparison evaluates Pick and Place with π0.5 under the same real-world protocol.
- Effect of Simulation SFT Initialization: Simulation SFT initialization makes RL optimization efficient, whereas omitting it leaves the policy near-trivial after more than three million interaction steps.The ablation compares real-only SFT initialization with full sim-real SFT initialization.
- Two-Stage Design and Real-Data Efficiency: Real-world supervision in both stages is necessary for deployment, and RL co-training improves over SFT co-training across tested mixture ratios and regularization weights.Removing Stage II anchoring causes domain drift, while removing supervision from both stages produces collapse; RL-Co also improves real-data efficiency.
6 Conclusion
RL-Co combines real and simulated demonstrations with reinforcement learning in simulation and real-world supervision, improving performance beyond supervised co-training alone. Experiments show gains in real-world success, robustness to distribution shifts, and data efficiency.
- RL-Co initializes with real and simulated demonstrations, then uses reinforcement learning with real-world supervision to preserve deployable behaviors.
- RL-Co outperforms real-only fine-tuning and SFT-based co-training across tasks and VLA models.
- RL-Co improves real-world success, robustness to distribution shifts, and data efficiency.
- Interactive simulation provides benefits that static imitation objectives alone cannot fully realize.
7 Limitations
The study is limited to tabletop manipulation with a single robot embodiment and does not evaluate heterogeneous sim-real settings. Although RL-Co improves real-world success, performance remains below 100%.
- The evaluation covers tabletop manipulation with one robot embodiment and excludes heterogeneous sim-real settings.
- RL-Co improves real-world success, but its performance remains below 100%.
A Detailed Problem Setup and Post-Training Objectives
The paper formulates real and simulated manipulation as related partially observable decision processes and defines a VLA policy that maps recent observations and language to future actions. It then describes supervised imitation and reinforcement-learning objectives, with RL-Co adding real-world supervision during simulation-based policy updates.
- A.1 Problem Formulation: Real and simulated tasks are modeled as POMDPs with domain-specific states, observations, transition dynamics, shared actions, language instructions, rewards, initial states, and discounting.
- A.1 Problem Formulation: The VLA policy conditions on recent observations and a language instruction to predict future actions over a fixed horizon.
- A.2 Post-Training Objectives: Post-training adapts a pretrained VLA policy using either expert demonstrations or online environmental interaction.
- A.2.1 Supervised Fine-Tuning (SFT): Supervised fine-tuning minimizes the discrepancy between predicted and expert action sequences, with the loss depending on the VLA architecture and action representation.
- A.2.2 Reinforcement Learning (RL) Fine-Tuning: Reinforcement learning fine-tuning maximizes expected discounted return from policy-sampled actions and environment transitions.
- A.2.2 Reinforcement Learning (RL) Fine-Tuning: RL-Co adds a real-world supervised fine-tuning objective during reinforcement-learning updates in simulation.
B Dataset Generation Details
The study combines teleoperated real demonstrations with large-scale simulated trajectories, evaluates four tabletop tasks using aligned robot and camera interfaces, and tests imperfect simulation anchored by real data.
- Real-world data: 20–50 successful teleoperated trajectories per task form the real-world dataset Dreal.Experts use a 3D Space-Mouse, and actions are recorded as end-effector delta-control commands.
- Simulation data: 1,000 successful simulated trajectories per task form Dsim after MimicGen replays real expert trajectories and prunes free-space motion.The simulation data are generated in ManiSkill from real-world seed behaviors, retaining task-relevant key stages.
- Tasks and setup: Four tasks require perception, language grounding, and control: Pick and Place, Push Cube via Instruction, Open Drawer, and Close Drawer.The environments use a Franka Panda with seven-degree-of-freedom end-effector delta control and fixed RGB observations.
- Generalization design: Real-world evaluation separates demonstrated and unseen object variations, including excluded cube colors and irregular-shaped objects absent from demonstrations.These splits support generalization evaluation beyond the demonstrated object set.
- Simulation fidelity: The simulator need not be photorealistic or a fully accurate digital twin because real-data supervision anchors the policy across visual differences.Object identities, materials, textures, lighting, and backgrounds are not precisely matched between domains.
- Reward design: Task rewards combine dense shaping and terminal success signals, with reward design varying by task and model.Pick and Place uses dense rewards for OpenVLA and sparse rewards for π0.5; drawer tasks issue terminal rewards of 5.0.
F Additional Dexterous Evaluation
The framework is also tested on Peg Insertion, a more dexterous task requiring precise contact and alignment, where RL-Co improves real-world success over SFT-based co-training.
- Task: Peg Insertion requires aligning a grasped 2 × 2 × 12 cm peg with a 2.25 × 2.25 cm square hole before insertion.The peg starts 1–5 cm above a horizontally perturbed hole, making the task more precision-demanding than the main tabletop experiments.
- Result: RL-Co raises real-world Peg Insertion success from 32.5% to 42.5% with the π0.5 backbone.The comparison uses 50 real demonstrations and 500 simulated trajectories under the same real-world evaluation protocol.
G Sim–Real Correlation During RL Co-Training
During RL co-training, simulation and real-world success generally improve together, although their relationship is imperfect because sim–real alignment is incomplete.
- Correlation: Real-world success generally increases as simulation success improves for π0.5 on Pick and Place and Open Drawer.Intermediate checkpoints are evaluated on the real robot while simulation success is recorded throughout Stage II RL co-training.
- Practical implication: Simulation convergence is used as a practical checkpoint-selection signal, while real demonstrations reduce domain drift during policy improvement.The positive correlation is useful but not perfect under imperfect sim–real alignment.
H.1 One-Stage RL Co-Training
Removing Stage I task-specific SFT prevents effective task acquisition, while static simulation-expert distillation helps but remains below direct RL-Co policy improvement.
- One-stage RL co-training: The one-stage variant achieves 0.0% real-world success and near-zero simulation success after several million interaction steps.It removes Stage I task-specific SFT and applies RL co-training directly to the pretrained VLA policy.
- Interpretation: Stage I SFT initialization provides the task-level behavior prior required for subsequent RL optimization.The result supports a two-stage design rather than replacing warm-starting with a single RL phase.
- Distillation comparison: The RialTo-style distillation baseline reaches 75.0% real-world success but remains below RL-Co.A simulation expert generates 2,000 trajectories that are then used for SFT co-training with 50 real demonstrations.