Source-linked AI summary
OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation
Zehao Yu, Jiakun Zheng, Weiji Xie, Jiyuan Shi, Chenyun Zhang, Chenjia Bai, Xuelong Li
TL;DR
Humanoid loco-manipulation lacks demonstration data that is simultaneously high-quality and scalable, while real-robot teleoperation remains costly and difficult to scale. OASIS reconstructs realistic assets, collects and augments teleoperated trajectories in simulation, and trains a hierarchical visuomotor policy. On a real humanoid, OASIS data collection runs up to 1.84× faster, and policies trained entirely on its data match or surpass real-robot-data policies under the same trajectory budget.
Problem
Humanoid loco-manipulation requires large-scale, high-quality demonstrations, but available real-robot teleoperation data is time-consuming, resource-intensive, and hard to scale.
Method
OASIS reconstructs object assets from real-world images, collects VR teleoperation trajectories in simulation, augments them through offline rendering, and trains a hierarchical whole-body policy.
Results
1.84× faster data collection than real-robot teleoperation; OASIS-trained policies match or surpass real-robot-data policies under the same trajectory budget.
Takeaways & Limitations
OASIS provides a practical and scalable alternative to real-robot teleoperation for zero-shot humanoid loco-manipulation transfer.
Takeaways & Limitations
Visual augmentation leaves trajectories unchanged, bounding motion diversity by operator demonstrations, while generated assets may have inaccurate geometry or physical parameters for visually complex objects.
Abstract
from arXiv · showhide
Recent progress in robot manipulation has been largely driven by learning from large-scale demonstrations. For humanoid robot loco-manipulation tasks, however, existing data sources force an unsatisfying tradeoff between trajectory quality and scalability. Real-world teleoperation provides the highest-quality trajectories but requires dedicated physical space and time-consuming scene resets. Simulation offers an alternative way out of this dilemma: it can produce clean, embodiment-aligned data at scale without any physical hardware. In this paper, we propose OASIS, a simulation-data-driven framework for humanoid loco-manipulation. OASIS automatically reconstructs realistic object assets from real-world images using a 3D generative model. Based on these assets, trajectories are first collected through teleoperation in simulation, and then augmented under diverse domain randomizations in a post-processing stage. With the resulting simulation data, we further design a hierarchical visuomotor policy for humanoid loco-manipulation. Extensive experiments on the real humanoid robot show that, under zero-shot deployment, the policy trained on our simulation data achieves higher success rates on most tasks than that trained on real-robot teleoperation data, owing largely to the broad lighting and environmental variations covered by our simulation rendering, which real-robot data fails to capture. The project page is available at https://oasis-humanoid.github.io/.
1 Introduction
OASIS addresses the shortage of scalable, high-quality humanoid loco-manipulation demonstrations with simulation-based collection and augmentation. Its simulation-trained hierarchical policy transfers zero-shot to real robots across multiple tasks.
- Humanoid loco-manipulation requires large-scale, high-quality demonstrations, but existing platforms still largely lack them.
- Real-robot teleoperation produces embodiment-aligned trajectories with action supervision, but is time-consuming, resource-intensive, and difficult to scale.
- OASIS reconstructs realistic 3D object assets from real-world images and uses them to build diverse, physically plausible simulation scenes at scale.
- OASIS separates real-time VR teleoperation in simulation from offline rendering and augmentation, reducing collection constraints while expanding visual training variation.The operator records robot and object state sequences, which are later replayed and rendered offline.
- A hierarchical visuomotor policy uses a Flow Matching high-level planner and a low-level controller to transfer simulation-trained behavior zero-shot to a real humanoid.The system also adapts to camera motion blur and background clutter.
2 Related Work
Humanoid loco-manipulation remains challenging because it combines coordinated locomotion, whole-body manipulation, and task-level reasoning while requiring substantial data supervision. Related work addresses these challenges through standardized policy-learning workflows, generalist VLA policies, simulation-based task and asset construction, and trajectory augmentation.
- Humanoid loco-manipulation combines coordinated locomotion, whole-body manipulation, and task-level reasoning, making both execution and data supervision challenging.
- Recent work standardizes humanoid policy learning into reproducible workflows to reduce the engineering burden.
- Generalist humanoid VLA policies are trained on heterogeneous mixtures of human videos, synthetic data, and teleoperated trajectories.
- Simulation is increasingly used as a scalable training source because real-robot data collection is costly.
- Foundation models can automate the construction of simulated tasks and assets, while trajectory augmentation replays a few demonstrations across new initial conditions.
- MimicGen established trajectory augmentation for tabletop manipulation, and DexMimicGen extended it to bimanual dexterous setups.
3 Method
OASIS builds humanoid loco-manipulation policies from scalable simulation data: realistic assets are generated from images, demonstrations are teleoperated and visually augmented, and a hierarchical policy transfers to a real robot.
- OASIS automates simulation scene construction and trains a hierarchical whole-body policy for zero-shot real-robot deployment.
- Simulation Scene Construction: Real-to-sim asset generation combines single-view object images, 3D mesh synthesis, and estimated physical dimensions and material properties.Generated meshes are rescaled, assigned density, friction, and restitution, and used to build physically plausible scenes.
- Simulation Data Collection: VR teleoperation collects humanoid manipulation trajectories in simulation while recording robot and interactive-object kinematic states plus retargeted reference motions.GMR retargets operator motions, and Teleopit drives the simulated humanoid during real-time rendering.
- Simulation Data Augmentation: Offline trajectory replay randomizes textures, lighting, and camera extrinsics to expand demonstrations into visually diverse training samples.The offline setting permits higher-fidelity Path-Tracing rendering without real-time teleoperation constraints.
- Hierarchical Policy: The hierarchical policy uses Flow Matching to predict reference motion commands from multimodal observations, while a low-level controller tracks target joint angles.The planner uses text, three-view images, and recent proprioception; reference commands provide a consistent history between simulation and deployment.
- Real-Robot Deployment: The system is deployed on a 29-DoF Unitree G1 with dexterous hands, using a 25 Hz high-level planner and a 50 Hz low-level controller.
4 Experiments
Experiments show that OASIS improves simulation data collection efficiency, benefits substantially from visual randomization, and transfers effectively to real humanoid robots under zero-shot deployment.
- 4.1 Data Collection Efficiency: OASIS collects successful trajectories significantly faster than real-robot teleoperation, with larger speedups on harder tasks.The comparison uses the same controller, operator, task, and number of successful trajectories; simulation resets are instantaneous and automatic.
- 4.1 Data Collection Efficiency: Simulation avoids fragile physical interactions, allowing operators to proceed without risking damage to manipulated objects.A real-robot screen-wiping collection damaged a monitor through excessive contact force, whereas simulated objects can simply be reset.
- 4.2 Ablations on Data Augmentation: Disabling all visual randomization nearly eliminates sim-to-real transfer, while the full combination outperforms every single-factor ablation.Lighting contributes most among individual factors, and texture, lighting, and camera-extrinsic randomizations are complementary.
- 4.2 Ablations on Data Augmentation: 15–20 renderings per trajectory approach success-rate saturation, leading OASIS to use 20 randomized environments.Additional renderings beyond this range provide tapering gains.
- 4.3 Effectiveness Of Simulation Data: With equal trajectory budgets, OASIS-only policies achieve real-robot success rates comparable to or higher than policies trained on real-robot data.The evaluation compares OASIS simulation data, real-robot data, and an equal mixture across multiple loco-manipulation tasks.
- 4.3 Effectiveness Of Simulation Data: Combining simulation and real-robot data outperforms either source alone under the same total trajectory budget.The paper attributes this gain to complementary supervision: simulation provides visual diversity, while real data captures interaction and perception characteristics.
5 Conclusion
OASIS combines realistic asset generation, simulation teleoperation, offline photorealistic rendering, and hierarchical policy learning for humanoid loco-manipulation. On the Unitree G1, it enables faster data collection and zero-shot transfer matching or surpassing real-robot training under equal trajectory budgets.
- 5 Conclusion: OASIS reconstructs simulation assets from real-world images and expands teleoperated demonstrations through offline photorealistic rendering.The pipeline produces visually diverse training samples without additional operator effort.
- 5 Conclusion: 1.84× faster data collection is achieved by OASIS than by real-robot teleoperation on the Unitree G1.The conclusion reports this as the maximum observed speedup.
- 5 Conclusion: Policies trained entirely on OASIS data transfer zero-shot to the real robot, matching or surpassing real-data policies under the same trajectory budget.These results support simulation as a practical and scalable alternative to real-robot teleoperation.
6 Limitations
OASIS transfers from simulation to real-world loco-manipulation zero-shot, but its data augmentation and asset reconstruction impose important limitations on motion diversity and sim-to-real fidelity.
- Motion diversity is bounded by operator demonstrations because augmentation randomizes visual appearance while leaving trajectories unchanged.Perturbing whole-body states can break balance, motivating physics-aware trajectory augmentation.
- Automatically generated assets may have inaccurate geometry and physical parameters for visually complex objects, widening the sim-to-real gap on contact-rich tasks.Improved asset reconstruction and physical-parameter calibration are identified as potential remedies.
A Material Density
Generated assets receive physical mass using mesh volume and category-level material density values listed in Table 3.
- Each generated asset is assigned mass from its mesh volume and a category-level material density.Table 3 lists the density values used in the experiments.
B Domain Randomization
OASIS applies domain randomization during offline rendering, with the randomized parameters documented in Table 4.
- Domain randomization is applied during offline rendering to diversify the simulation data.Table 4 defines the randomized parameters, including uniform and Bernoulli distributions.
C Real-to-Sim Asset Generation Details
OASIS uses a vision-language model to estimate physical attributes from a reference image and category description, then parses the response into each asset’s physical properties.
- Qwen3-VL is prompted with a reference image and object category to estimate length, width, height, and material in JSON format.The prompt requests dimensions in centimeters and a material category.
- The model’s JSON output populates the corresponding 3D asset’s physical dimensions and material category.
C.2 Dimension Accuracy Evaluation
This evaluation compares Qwen3-VL’s predicted object dimensions with caliper-based ground-truth measurements across five real-world objects.
- Table 5 reports predicted and measured dimensions for five real-world objects, together with each prediction’s relative error.
D Ablation on Curriculum-based Rollout
This ablation evaluates curriculum-based rollout by comparing planner training with and without rollout across four manipulation tasks. Training without rollout produces compounding long-horizon errors and consistently lower success rates.
- The ablation compares planners trained exclusively on ground-truth history with planners using curriculum-based rollout.
- Both variants are evaluated on four manipulation tasks using per-task success rates and an overall average.
- Training without rollout causes compounding errors over long horizons, resulting in consistently lower success rates across all tasks.