PaperlayerView exact revision on arXiv

Source-linked AI summary

SimVLA: Zero-Shot Sim-to-Real VLA Learning for Mobile Manipulation

Kyoungin Baik, Youngwoon Lee

arXiv:2610.11248v1cs.RO

TL;DR

Robot-action data are costly and limited in diversity, while simulation’s potential for sim-to-real VLA learning in mobile manipulation remains underexplored. SimVLA trains on synthetic action, visual-language, and policy-rollout data, then transfers zero-shot to real environments, outperforming policies trained on 50 in-domain demonstrations in its evaluation settings.

  • Problem

    Robot-action data are expensive to collect and limited in task and environmental diversity, while simulation-based sim-to-real VLA learning for mobile manipulation remains underexplored.

    Source: §1 · p. 2

  • Method

    SimVLA pre-trains on synthetic SimAction trajectories and SimVQA supervision, then post-trains on SimAction mixed with SimDeploy policy rollouts.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 4

  • Results

    Across five tasks and three real-world environments, simulation-trained SimVLA outperforms policies trained on 50 in-domain real-world data in the reported evaluation settings.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    The results suggest that combining action, visual-language, and deployment-oriented supervision in simulation can support zero-shot sim-to-real mobile manipulation.

    Source: §1 · p. 2

    The evaluation focuses on rigid-object manipulation in kitchen-like environments and a subset of embodiments and architectures.

    Source: §6 · p. 11

Abstract

from arXiv · show

Large-scale, diverse datasets have driven the success of LLMs and VLMs. But VLAs for robotics remain limited by the cost and complexity of real-world data collection. While simulation offers a scalable alternative, its potential for sim-to-real VLA learning in mobile manipulation remains largely underexplored. We introduce SimVLA, an end-to-end framework that trains VLAs entirely on synthetic simulation data without teleoperation for mobile manipulation. SimVLA is first pre-trained on two complementary simulation-derived datasets: SimAction, a large-scale robot action dataset spanning 35 diverse mobile manipulation tasks, generated by composing atomic skills, and SimVQA, which leverages privileged simulator state to provide spatial, geometric, and subtask-level visual-language supervision. We further post-train SimVLA on a mixture of SimAction and SimDeploy, a dataset collected from policy rollouts across diverse simulated environments. We evaluate SimVLA on tasks including restocking, pouring, and cleaning, and show zero-shot transfer to real-world mobile manipulation, including real home environments. SimVLA outperforms policies trained on 50 in-domain real-world demonstrations, suggesting that simulation can enable scalable sim-to-real mobile manipulation. We further demonstrate the value of multiple complementary forms of supervision for effectively leveraging simulation in VLA training.

1. Introduction

SimVLA addresses the costly, limited diversity of real-world robot-action data by using simulation for action, visual-language, and deployment-oriented supervision. It transfers zero-shot to real environments and outperforms a policy trained on 50 real-world demonstrations in the demonstration environment.

1. Introduction

  • Robot action data are costly to collect, embodiment-specific, and limited in task and environmental diversity, especially for long-horizon mobile manipulation.Mobile manipulation also couples navigation and manipulation under partial observability, making broad coverage through teleoperation difficult.Source: §1 · p. 2
  • The framework combines SimAction action trajectories, SimVQA visual-language supervision, and SimDeploy policy rollouts to train mobile-manipulation VLAs entirely in simulation.SimAction spans 35 tasks; SimDeploy exposes the model to policy rollouts across diverse simulated environments.Source: §1 · p. 2
  • SimVLA transfers to unseen real-world environments without real-world training data, outperforming a policy fine-tuned on 50 real-world demonstrations in the demonstration environment.It also maintains robust performance in a pantry and a real home, where both baselines degrade sharply.Source: §1 · p. 2
  • Table 1 compares simulation data-generation methods using visual-language supervision and policy rollouts as comparison dimensions.Its caption defines visual-language supervision as annotations beyond action trajectories and policy rollouts as trajectories collected from a learned policy in simulation.Source: Table 1 · p. 3

2. Related Work

Prior VLA and sim-to-real work faces costly data collection and a persistent simulation-to-reality gap, with mobile-manipulation evaluations still limited. SimVLA differentiates itself by using simulation for action, visual-language, and deployment-oriented supervision and evaluating coupled base–arm control across tasks and real environments.

Data for Training VLAs; Synthetic Data Generation

  • Teleoperated robot-action data are expensive to scale, while augmenting source demonstrations can leave generated data constrained by those demonstrations.Motion-planning approaches generate synthetic data without teleoperation, but existing approaches largely treat simulation as a source of action data.Source: §2 · p. 3

Sim-to-Real

  • Zero-shot sim-to-real remains challenging because of the simulation-to-reality gap; prior methods often combine system identification with domain randomization.SimVLA combines hybrid system identification with broad domain randomization.Source: §2 · p. 3

Synthetic Data Generation; Sim-to-Real

  • Unlike approaches that mainly use simulation to scale action data, SimVLA combines action trajectories, simulator-derived visual-language supervision, and policy-rollout data.SimAction composes atomic skill APIs, SimVQA uses privileged simulator state, and SimDeploy exposes policy-induced states.Source: §2 · p. 3

Mobile Manipulation

  • SimVLA targets end-to-end VLA learning with coupled base–arm control across 35 practical tasks and evaluates zero-shot transfer in three unseen real-world environments.Prior mobile-manipulation work had limited task diversity and real-world deployment performance in the described comparison.Source: §2 · p. 4

3. SimVLA: Synthetic Data Generation and VLA Training

SimVLA builds synthetic mobile-manipulation data from procedurally varied kitchens and trains on complementary action, visual-language, and deployment-oriented supervision. Its two-stage pipeline pre-trains on SimAction and SimVQA, then post-trains on SimAction mixed with SimDeploy.

3.1. Procedural Kitchen Scenes

  • Procedurally generated scenes vary five kitchen layouts, furniture and object arrangements, cabinet configurations, materials, and object shapes.The five layouts are island, L-shaped, peninsula, single-wall, and U-shaped kitchens.Source: §3.1 · p. 4

3.2. SimAction: Synthetic Action Data

  • SimAction contains approximately 200K trajectories spanning 35 mobile-manipulation tasks across 100 procedurally generated houses, created without teleoperation through skill composition and motion planning.Trajectories pair observations and language instructions with continuous arm end-effector poses and base velocities.Source: §3.2 · p. 4
  • Composable skill APIs generate reusable long-horizon task specifications, while a visual search skill lets the policy locate relevant objects under partial observability.The APIs include skills such as grasping, placing, navigating, and searching; a GUI supports optional grasp preferences and user-defined skill sequences.Source: §3.2 · pp. 4–5
  • Hybrid system identification constrains both joint-space behavior and end-effector tracking before collision-free motion plans generate trajectories for three robot embodiments.The trajectories are computed from subtask goals using cuRobo.Source: §3.2 · p. 5

3.3. SimVQA: Simulation-Derived Visual-Language Supervision

  • SimVQA converts simulator ground truth into about 1M question-answer samples on object locations, spatial relations, geometry, robot state, and subtask progress.This provides intermediate supervision between perception and action without the costly post-processing needed for comparable annotations from real-world demonstrations.Source: §3.3 · p. 5

3.4. SimDeploy: Deployment-Oriented Rollout Data

  • SimDeploy collects 22K policy-rollout trajectories across diverse simulated environments to capture policy-induced states, including varied viewpoints and scene layouts.The authors report that post-training on SimDeploy improves robustness and generalization to unseen real-world environments.Source: §3.4 · p. 5

3.5. SimVLA: Two-Stage VLA Training

  • Pre-training jointly predicts continuous and tokenized SimAction actions and SimVQA answers; post-training optimizes action flow matching on SimAction mixed with SimDeploy.The model allows gradients to update the VLM backbone during post-training after stopping action-head gradients into the backbone during pre-training.Source: §3.5 · p. 6, Figure 2 · p. 6

4. Experiments

Experiments span simulated and real-world mobile manipulation, comparing zero-shot transfer with demonstration-trained policies and testing how pre-training, supervision sources, and rollout data affect generalization.

4.1. Simulation Experiments

  • Simulation experiments compared SimVLA with MolmoB0T on RB-Y1 across held-out SimVLA kitchens and MolmoSpaces, and also evaluated eight long-horizon tasks with Anubis.The supplied passages do not report the results of the eight-task Anubis evaluation.Source: §4.1 · p. 6

4.2. Real-World Experiments

  • Real-world evaluation covered five long-horizon tasks across a mock kitchen, pantry, and real home, with ID, object/furniture-repositioned, and randomized-background/lighting conditions in the mock kitchen.Task progress was measured as the percentage of predefined subgoals completed; the study reports 2,120 rollouts.Source: §4.2 · pp. 6–7

4.3. Can SimVLA Match Policies Trained with In-Domain Demonstrations?

  • SimVLA achieved 54.4% average task progress and outperformed the policy fine-tuned on 50 in-domain demonstrations; increasing demonstrations to 200 gave limited OOD gains.The comparison was made in the evaluated settings, including OOD-pos and OOD-vis conditions.Source: §4.3 · p. 8

4.4. Does SimVLA Need Pre-Training?

  • Removing pre-training substantially degraded performance; the 𝜋0.5 policy trained on 200 demonstrations fell from 67.8% task progress in mock kitchen-ID to 21.3% in the pantry and 29% in the real home.SimVLA’s pre-training data came entirely from simulation.Source: §4.4 · p. 8

4.5. Pre-training Data Ablation

  • Removing either SimVQA or Tokenized SimAction degraded performance across all five tasks; on 1,369 held-out SimVQA pairs, the full model achieved 0.06 m median 3D position error and 0.61 cross-view IoU.Without SimVQA, the model often failed because of imprecise grasping.Source: §4.5 · p. 9

4.6. Post-training Data Ablation

  • SimDeploy substantially improved sink and drawer tasks by adding varied layouts, viewpoints, and retries, but gave smaller gains for pouring, where SimAction randomization already covered object-level variation.Sink and drawer tasks required broader scene-level variation.Source: §4.6 · p. 9

4.7. What Makes Mobile Manipulation Hard in OOD Situations?

  • In OOD-pos, demonstration-trained policies mostly failed to grasp drawer handles after small trajectory errors, while SimVLA’s randomized viewpoints and spatial configurations yielded more robust performance.The authors identify limited trajectory coverage as a key OOD-generalization bottleneck.Source: §4.7 · p. 10

4.8. How does SimVLA compare to existing zero-shot sim-to-real mobile manipulation baselines?

  • SimVLA maintained stable zero-shot performance on MolmoSpaces, while MolmoB0T degraded in held-out SimVLA kitchens; with unseen targets, MolmoB0T achieved 0% success and SimVLA used active vision.None of the environments were included in training.Source: §4.8 · p. 10

5. Conclusion

SimVLA trains mobile-manipulation VLAs entirely from simulation, combining diverse synthetic data and complementary supervision. Across five tasks and three real-world environments, it outperforms policies trained on 50 in-domain real-world data in the reported evaluation settings.

5. Conclusion

  • SimVLA, trained purely in simulation, outperforms policies trained on 50 in-domain real-world data in the evaluation settings.It transfers without environment-specific data to practical settings including a pantry and a real home.Source: §5 · p. 11
  • SimVLA combines procedural kitchen generation, hybrid system identification, domain randomization, and SimVQA supervision to create diverse synthetic training data without teleoperation.SimDeploy rollouts expose the policy to scene-level diversity and policy-induced states beyond procedural demonstrations.Source: §5 · p. 11
  • Experiments span five tasks and three real-world environments, and comparison with a current sim-to-real VLA baseline demonstrates generalization across mobile-manipulation settings.The paper presents simulation as a source of action trajectories, visual-language supervision, and deployment-oriented rollout data.Source: §5 · p. 11

6. Limitations & Future Work

The evaluation leaves several boundaries unresolved: domain-randomization factors are not isolated, tasks focus on rigid objects in kitchen-like environments, and only a subset of embodiments and architectures is evaluated.

6. Limitations & Future Work

  • The study does not isolate the contributions of individual domain-randomization factors.The authors identify this as a limitation of the evaluation.Source: §6 · p. 11
  • Tasks focus on rigid-object manipulation in kitchen-like environments; deformable objects, dexterous manipulation, and broader household settings remain future work.Source: §6 · p. 11
  • Evaluation covers only a subset of embodiments and architectures, so broader evaluation is needed to assess generality.The authors characterize SimVLA as a simulation-based data-generation framework, not a cross-embodiment transfer method or a new VLA architecture.Source: §6 · p. 11
  • SimAction generation relies on human-defined skill APIs; using LLMs to expand skills as code-based policies is proposed as future work.Source: §6 · p. 11

A. Kitchen Scene

SimVLA’s kitchen-scene pipeline varies layouts, objects, placements, and materials, while its skill APIs support long-horizon task composition and diverse search and grasp demonstrations.

A. Kitchen Scene

  • Five kitchen layouts—peninsula, L-shaped, U-shaped, island, and single-wall—vary navigation and manipulation challenges, with randomized furniture, objects, cabinets, and materials.Examples include island navigation around a central counter and precise base positioning across U-shaped surfaces.Source: A. Kitchen Scene · p. 18, Figure 8 · p. 18
  • Task-relevant objects are sampled across varied shapes, sizes, scales, poses, and materials to increase geometric and visual diversity.The paper notes that policies trained on narrow object sets tend to overfit and fail under shape variation; Figure 9 shows 40 bowl variants.Source: A. Kitchen Scene · p. 18, Figure 9 · p. 18
  • The scene-generation GUI lets users choose layouts, categories, and semantic placement locations, review object lists, and generate scenes with randomized instances, poses, and materials.Source: A. Kitchen Scene · p. 18, Figure 10 · p. 18

B. Skill API

  • Atomic skill APIs, including grasping, placing, pouring, opening, closing, navigating, and searching, are registered for composition into long-horizon task definitions.The grasp example samples a pose, moves through intermediate and final poses, then closes the gripper.Source: B. Skill API · p. 19
  • Search demonstrations use ground-truth poses and collision geometry during data generation, while the learned policy searches from visual observations alone at inference.The scripted trajectory rotates, approaches via perturbed collision-checked waypoints, and varies final heading; grasping also uses up to 200 BODex candidate poses for optional human selection.Source: B. Skill API · p. 19, Figure 11 · p. 19

C. Hybrid System Identification

Hybrid system identification calibrates simulated dynamics against real-robot tracking, while SimAction defines the task data and domain randomization broadens training conditions.

C. Hybrid System Identification

  • Hybrid identification jointly minimizes end-effector position and rotation errors and joint-space MSE, with 4096 parallel Isaac Lab environments reducing calibration from hours to minutes.Candidates are evaluated against real-robot tracking signals on calibration sequences and accepted only if all three errors stay below their thresholds across timesteps.Source: C. Hybrid System Identification · p. 20, Figure 13 · p. 20
  • 6.3× lower mean EEF error (83.4 → 13.3 mm) and 3.0× lower maximum error (120.7 → 40.6 mm) result from hybrid identification over default gains.Hybrid is best on both metrics, but gains only 1.4 mm in mean error over joint-space identification and is within noise of EEF-space identification; Table 3 reports EEF position discrepancy over ten held-out trajectories on both arms.Source: C. Hybrid System Identification · p. 21, Table 3 · p. 21

D. SimAction

  • SimAction contains 35 tasks: 22 long-horizon compositions and 13 atomic skills, with each atomic skill also released as a standalone task.Each task provides three RGB streams at 240×320 resolution and 20 fps, proprioception, initial robot pose, and a 23-D action space.Source: D. SimAction · p. 22

E. Domain Randomization

  • Domain randomization spans camera, kitchen-scene, object, robot-dynamics, and lighting conditions to expose policies to physical and visual discrepancies between simulation and reality.The setup is summarized in Table 5.Source: E. Domain Randomization · p. 23, Table 5 · p. 23

F. SimVQA

SimVQA is automatically generated from simulator state to provide visual-language supervision for geometric and subtask-level reasoning. Its roughly one million examples span tasks from planning to gripper-state queries.

F. SimVQA

  • SimVQA uses ground-truth simulator state to generate precise spatial labels that would be prohibitively expensive to obtain from real-world data.Source: F. SimVQA · p. 24
  • The dataset contains approximately 1M visual question-answering samples.Source: F. SimVQA · p. 24
  • Its templates cover planning, subtask sequencing, object detection, 3D spatial reasoning, cross-view correspondence, reachability, graspability, and gripper-state queries.Source: F. SimVQA · p. 24, Table 6 · p. 24

G. SimVLA Training

SimVLA uses a two-stage pipeline with a pre-trained VLM backbone and flow-matching action expert. Pre-training combines action and SimVQA supervision while insulating the backbone from flow-loss gradients; post-training adapts the model on mobile-manipulation rollouts and tasks.

G. SimVLA Training

  • Pre-training jointly applies flow matching on continuous actions, autoregressive FAST-action prediction, and autoregressive SimVQA-answer prediction in one forward pass.The noisy-action expert predicts a vector field toward the clean action trajectory, conditioned on the prefix and interpolation time; the action horizon is 50 with dimension 23.Source: G. SimVLA Training · p. 25
  • Knowledge insulation stops flow-loss gradients at the VLM backbone and blocks cross-attention between continuous-action tokens and FAST/VQA tokens.FAST and VQA losses still update the backbone during pre-training.Source: G. SimVLA Training · p. 25
  • Post-training drops FAST and VQA losses, lifts the flow-loss stop-gradient, and fine-tunes on mobile-manipulation tasks matching evaluation.The post-training data combines the original continuous-action corpus with deployment data from policy rollouts across diverse simulated environments.Source: G. SimVLA Training · p. 25
  • The warmed-up action expert allows backbone adaptation during post-training while avoiding instability from jointly training a randomly initialized expert with pre-trained VLM representations.Source: G. SimVLA Training · p. 26
  • The reported experiments required approximately 1,272 GPU-hours, with additional compute spent on preliminary experiments, tuning, failed runs, and debugging.Source: G. SimVLA Training · p. 26

H. Experimental Setup

The experiments evaluate SimVLA on an Anubis mobile manipulator using task-progress and binary-success measures across simulation and real-world settings. SimVLA improves on the 50-demonstration baseline across the reported mock-kitchen comparisons, while results vary by distribution shift and task.

H. Experimental Setup

  • The Anubis platform has two 6-DoF arms, wrist-mounted cameras, a central adjustable ego-centric camera, and a three-wheel omni-directional base.Source: H. Experimental Setup · p. 28, Table 10 · p. 28, Figure 14 · p. 28
  • Task progress uses discrete labels to measure completion beyond binary success; the evaluation reports the percentage of predefined subgoals completed.Source: H. Experimental Setup · p. 28
  • Across eight long-horizon simulation tasks, Put mug and bowl to sink succeeds in 19/50 rollouts; its two navigation and pick-and-place sequences create more opportunities for failure.Tasks span navigation, grasping, placement, articulated drawer operation, and bimanual pouring; evaluation uses five kitchen setups and 50 rollouts per task.Source: H. Experimental Setup · p. 29, Table 12 · p. 29
  • SimVLA exceeds the 50-demonstration 𝜋0.5 baseline in task progress across all mock-kitchen ID tasks, and is best on task progress across OOD-Pos tasks.It has comparable or higher binary success on ID and best-or-comparable binary success on OOD-Pos; on OOD-Vis it is best on Tasks 1, 3, 4, and 5, while 𝜋0.5 fine-tuned with 200 demonstrations is best on Task 2.Source: Table 13 · p. 30, Table 14 · p. 30, Table 15 · p. 30
  • Removing either SimVQA or TokSimAction lowers task progress on every evaluated mock-kitchen ID task relative to full SimVLA.Removing SimVQA mainly degrades control through less precise grasping.Source: Table 16 · p. 31
  • In the pantry and real home, evaluations cover pantry Tasks 1, 4, and 5 and real-home Tasks 1 and 3, with 20 and 10 trials per task, respectively.Source: Table 17 · p. 31

Related papers

Loading 2610.11248v1…