Learning to Stack: Cube-Stacking Imitation Learning from Virtual Reality Demonstrations

• Long-horizon demonstrations make data collection costly because trials require repeated object resets, robot repositioning, and scene recovery after failures.These costs are especially apparent in cube-stacking, which depends on a clean initial scene.Source: §I · p. 1
• Simulation can reduce collection costs through instant resets, replay, and post-collection sensor changes, while Isaac Lab Mimic supports augmentation of small human demonstration sets [6].Isaac Gym and Isaac Lab combine accelerated physics simulation with robot-learning tooling,.Source: §I · p. 1
• The work focuses on a custom 5-DoF arm and argues that a robust, reusable virtual-demonstration method is needed for compact manipulation platforms.Smaller manipulators lower hardware cost and setup time, supporting rapid iteration on teleoperation and data pipelines.Source: §I · p. 2
• Figure 1 presents the intended system flow from immersive teleoperation through replay validation to synthetic dataset expansion.Source: Figure 1 · p. 1, §I · p. 2

Paper Contributions

• The paper contributes a VR teleoperation pipeline for cube-stacking demonstrations with LeRobot-style arms in Isaac Sim and Isaac Lab.Source: Paper Contributions · p. 2
• Its replay and retargeting process verifies demonstrations, converts SE(3) inverse-kinematics actions to joint-space actions, and supports later sensor or state re-rendering.Source: Paper Contributions · p. 2
• The work compares virtual and real-world collection efficiency, expands data with Isaac Mimic, and evaluates a dual-camera ResNet18-LSTM policy in Isaac Sim.Source: Paper Contributions · p. 2

Article Structure

• The six sections move from related work and pipeline methods through experimental setup and results to the conclusion and future work.Source: Article Structure · p. 2

II. RELATED WORK

• The review frames VR telemanipulation, imitation learning, and simulation-based data generation as established robotics topics.Source: §II · p. 2

1) VR Teleoperation and Telepresence:

• Prior work includes RoboTurk for scalable remote demonstrations, avatar-style telepresence for remote manipulation, and Open-TeleVision for immersive feedback during long-horizon data collection –.Source: 1) VR Teleoperation and Telepresence · p. 2
• The cited VR efforts were not centered on rapid dataset collection for fixed-base manipulation in simulation, the focus distinguished in this paper.Source: 1) VR Teleoperation and Telepresence · p. 2

2) Imitation Learning and Shared Demonstration Pipelines:

• Robomimic highlights dataset quality and algorithm choice, ALOHA Unleashed explores scaling accessible-hardware collection, and LeRobot promotes shared hardware and data formats –[4].Deep visuomotor policy work motivates learning manipulation policies directly from camera observations.Source: 2) Imitation Learning and Shared… · p. 2

3) Isaac-Based Data Generation:

• Isaac Lab Mimic records and annotates subtasks, replays and transforms demonstrations relative to task objects, then generates trajectories for Robomimic behavior cloning.The paper applies this approach to a custom small arm, immersive VR teleoperation, and cube-stacking data collection.Source: 3) Isaac-Based Data Generation · p. 2

III. METHODOLOGY

• The methodology section outlines a VR pipeline for generating cube-stacking demonstrations and preparing them for imitation-learning experiments.It points to further implementation detail on the interface, replay, and synthetic-data generation.Source: §III · p. 2

A. System Overview

• The manipulation environment integrates a custom 5-DoF arm into Isaac Sim and Isaac Lab.Source: A. System Overview · p. 2
• The small arm is inexpensive, easy to iterate on, and representative of benchtop hardware research groups can deploy.Source: A. System Overview · p. 3
• HTC Vive Pro 2 and Manus Quantum gloves provide VR control; OpenXR links the hardware to simulation, and Pink inverse kinematics maps SE(3) end-effector targets to joint motion.Task-space teleoperation was chosen as more intuitive for grasping and placement than direct joint-level control.Source: A. System Overview · p. 3

B. Cube-Stacking Task and Observations

• Cube positions are randomized at the start of each demonstration, which requires placing red on blue and then green on red.Source: B. Cube-Stacking Task and Observations · p. 3
• Two RGB views—a fixed camera behind the robot and a gripper-mounted camera—are added during replay, alongside randomized scene appearance.Replay can also render updated model-state observations or actions.Source: B. Cube-Stacking Task and Observations · p. 3

C. Replay, Retargeting, and Dataset Generation

• Recorded task-space motion and scene evolution are replayed for reproducibility, then inverse-kinematics pose actions are converted to joint-space sequences for export.The reusable demonstrations support changed sensors, state observations, and action representations; Isaac Mimic also creates success-filtered trajectories by transforming and stitching subtasks under randomized initializations.Source: C. Replay, Retargeting, and Dataset… · p. 3, Figure 3 · p. 3

• Demonstration throughput is one evaluation dimension.Source: §IV · p. 3
• Dataset expansion is another evaluation dimension.Source: §IV · p. 3
• The evaluation also includes an initial simulation-only policy test.Source: §IV · p. 3

A. Manual Demonstration Collection

• Approximately 200 virtual demonstrations were collected in 30 minutes, compared with approximately 45.0 in the same time using the real-world workflow.The reported collection rates were approximately 6.67 demonstrations per minute virtually and 1.5 per minute in the real world.Source: A. Manual Demonstration Collection · p. 3

C. Policy Training and Simulation Evaluation

• 10% full-task success was achieved in 30 randomized simulation trials by the trained policy.Full success required placing red on blue, then green on red; Table I summarizes subtask success rates.Source: Table I · p. 4, C. Policy Training and Simulation… · p. 4

B. Synthetic Demonstration Generation

• 100 synthetic demonstrations were added to the 200 manually collected examples, yielding 300 total after approximately three days of computing.Synthetic generation was slower in wall-clock time than manual VR teleoperation but required no continuous operator involvement.Source: B. Synthetic Demonstration Generation · p. 4

V. RESULTS AND DISCUSSION

• 200 virtual demonstrations in 30 minutes versus 45 real-world demonstrations in the same time yielded an approximately 4.4× increase in demonstrations per operator time.The improvement mainly came from instant scene resets and removing repeated manual task-state recovery.Source: §V · p. 4
• Replay allows the same demonstrations to support multiple camera or observation variants, reducing the risk of recollecting data when policy inputs change.The passage identifies Figure 2 as illustrating the replay-centered structure.Source: §V · p. 4
• Isaac Mimic added 100 demonstrations over approximately three days through unattended expansion, complementing manual VR collection despite slower wall-clock time.The supplied table caption identifies Table II as a demonstration collection summary but provides no cell values.Source: §V · p. 4, Table II · p. 4
• The policy completed the full stack in 3 of 30 randomized simulation trials, a 10% success rate, with color-order and gripper-alignment errors among its main failures.It often selected the closest cube rather than the required next cube, and alignment errors caused cubes to jam in the pinch region.Source: §V · p. 4
• The results support the pipeline as a practical first step for small-arm imitation-learning data, not yet as a robust cube-stacking policy.Evaluation was only in simulation, while synthetic trajectory generation depends on seed-demonstration quality and replay consistency.Source: §V · p. 4

VI. CONCLUSION

• The pipeline combines immersive SE(3) teleoperation, replay-based validation, task-space-to-joint-space retargeting, dual-camera observation generation, and Isaac Mimic expansion.It was developed for a custom 5-DoF arm in Isaac Sim and Isaac Lab.Source: §VI · p. 4
• Approximately 200 virtual demonstrations were collected in 30 minutes, compared with 45 for a comparable real-world process, and 100 additional synthetic demonstrations were generated.The additional samples were generated from the 200 manual trajectories.Source: §VI · p. 4
• A dual-camera ResNet18 and LSTM policy achieved 10% full-task success across 30 randomized simulation trials, with failures mainly involving cube-color confusion and gripper alignment.The paper identifies improving policy reliability, expanding dataset variation, and deployment to additional small manipulation platforms as future work.Source: §VI · p. 4