VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation

• Autonomous loco-manipulation with onboard perception remains rare in real humanoids, while prior systems often isolate locomotion, fixed-base manipulation, or rely on teleoperation or non-onboard sensors.Source: §1 · pp. 1–2
• Humanoid teleoperation data may be prohibitively expensive, while GPU simulation can generate orders of magnitude more data at low marginal cost.Source: §1 · p. 2
• The paper presents a technical recipe for making RGB-based humanoid loco-manipulation work in practice, rather than proposing a new RL or sim-to-real algorithm.Source: §1 · p. 2
• VIRAL trains a privileged full-state RL teacher in simulation, then distills it into an RGB-and-proprioception student using online DAgger and behavior cloning.Source: §1 · p. 2
• Visual randomization and high-fidelity tiled rendering are paired with real-to-sim alignment of dexterous hands and cameras to support zero-shot transfer.Source: §1 · p. 2
• VIRAL’s real-world experiments report high success near expert teleoperation and generalization to spatial and scene variations.Source: §1 · p. 2

2. Key Elements of VIRAL

• VIRAL trains a privileged-state RL teacher, distills it into an RGB-and-proprioception student, then randomizes and aligns simulation before zero-shot deployment.The student learns from a mixture of online DAgger and behavior cloning.Source: §2 · p. 3

2.1. Key Elements of Teacher Training

• The teacher outputs delta commands through a WBC controller, with staged rewards for walking, placing, grasping, and turning.The WBC command interface includes locomotion, upper-body, and finger actions.Source: §2.1 · p. 3
• 200 teleoperated simulation demonstrations seed episode resets with rewarding robot, object, and table states, exposing the teacher to grasping and placement poses early.The paper describes this reference-biased initialization as essential to training humanoid loco-manipulation.Source: §2.1 · p. 4, Figure 4 · p. 4

2.2. Key Elements of Student Training

• The student combines teacher- and student-induced observations for DAgger and behavior cloning, and fuses RGB features with proprioception in its policy input.The student uses an image encoder; the authors also evaluate single-step and history-aware policy heads.Source: §2.2 · p. 4
• A customized distributed simulation system supports visual student training across GPUs, with the authors identifying GPU scaling as critical in ablations.The implementation is described as enabling near-linear scaling across multiple GPUs and compute nodes.Source: §2.2 · p. 5

2.3. Key Elements of Sim-to-Real Transfer

• Sim-to-real transfer combines dexterous-hand system identification, camera-extrinsics alignment, and visual randomization over image quality, lighting, materials, and viewpoint.Training also randomizes camera latency and extrinsics to account for delays and hardware-induced viewpoint differences.Source: Figure 3 · p. 4, §2.3 · p. 5

3.1. Robustness

• 54 of 59 consecutive real-world trials succeeded during repeated walking, placing, grasping, and turning between tables.The evaluation used a 29-DoF Unitree G1 with 7-DoF three-finger dexterous hands.Source: §3.1 · p. 5, §3 · p. 5
• VIRAL reached 20.2 s cycle time, versus 21.4 s and 100% success for the expert teleoperator; the non-expert achieved 73% success with slower execution.All conditions used the same HOMIE policy for a near-apple-to-apple comparison.Source: §3.1 · pp. 5–6

3.2. Generalization

• VIRAL completed the task without additional tuning across variations in tray and robot start positions, table height and type, lighting, tablecloth, and object category.The real-world generalization figure depicts variations in these conditions.Source: §3.2 · p. 6, Figure 8 · p. 6

4.1. Reference State Initialization for RL

• Nearly 95% success with reference state initialization contrasts with below 10% without it.The full VIRAL teacher combines reference state initialization with delta actions.Source: §4.1 · p. 6, Figure 9 · p. 6

4.2. Delta versus Absolute Action Space

• Only the delta-action teacher reliably solved the task; the absolute-action variant failed to reach high success.The ablation compares training rewards and success rates for the full method and variants without demonstration resets or delta actions.Source: §4.2 · p. 6, Figure 9 · p. 6

4.3. Vision Backbone

• DINOv3 yielded stronger visual representations, faster convergence, and higher task success than the alternative vision backbone.Figure 10 reports student training loss and success rate.Source: §4.3 · p. 6, Figure 10 · p. 7

4.4. DAgger versus BC Visual Distillation

• The α = 0.5 DAgger–BC mixture improved deployment success over pure BC, despite slightly slower optimization.Pure BC reduced loss quickly but produced a brittle policy that performed poorly in Isaac-to-MuJoCo and real-world evaluations.Source: §4.4 · p. 7, Figure 11 · p. 7

4.5. History Architecture

• History-aware models consistently outperformed the single-step baseline, with longer temporal windows adding gains when resources allowed.The comparison includes a feed-forward history model and an LSTM under different history lengths.Source: §4.5 · p. 7, Figure 12 · p. 7

4.6. Visual Randomization

• Removing all visual randomization reduced normalized success to 0.649, a 35.1% decrease; removing any single component also degraded performance.The tested dominant components were material, dome-light, and camera-extrinsics randomization; other factors provided smaller gains.Source: §4.6 · p. 7, Figure 13 · p. 7

4.7. Scaling Compute for Teacher Training

• With 8–16 GPUs, teacher success consistently exceeded 90%; with 1–2 GPUs, training plateaued far below the desired range.Scaling from 1 to 16 GPUs also accelerated learning, with larger parallel batches broadening state-space coverage per unit wall time.Source: §4.7 · p. 7, Figure 14 · p. 7

4.8. Scaling Compute for Student Training

• Larger-scale student training reduces success-rate variance, especially early when the policy is more sensitive to distribution shift.The reported scale-up also yields richer, more diverse state coverage, which the authors associate with improved robustness.Source: §4.8 · p. 8
• More GPUs accelerate student convergence, smooth optimization, and slightly improve final success, making substantial compute a practical requirement for reliable visual distillation.The student scaling study varies training from 1 to 64 GPUs.Source: §4.8 · pp. 7–8, Figure 15 · p. 8

4.9. Object generalization

• Training on ten distinct objects yields substantially better object generalization than training on a cylinder alone, evaluated across the same ten objects.The study reports normalized success rates for the grasping subtask.Source: §4.9 · p. 8, Figure 16 · p. 8

Sim-to-Real for Locomotion

• Proprioceptive sim-to-real locomotion is robust but lacks environmental awareness; depth or LiDAR supports terrain geometry, while RGB-language approaches rely on high-latency VLA models.VIRAL instead uses compact RGB-driven visuomotor policies for real-time, goal-conditioned locomotion.Source: §5 · p. 8

Sim-to-Real for Manipulation

• Visual domain randomization and teacher-student distillation have advanced manipulation sim-to-real, but prior work largely targets tabletop settings.The cited line includes dexterous-hand tasks such as object reorientation and Rubik’s Cube solving.Source: §5 · p. 8

Sim-to-Real for Loco-Manipulation

• Prior loco-manipulation systems use modular or end-to-end low-level controllers, while task-level imitation or VLA approaches require large real-world datasets and often lack robustness.VIRAL trains an RGB-driven end-to-end policy entirely in simulation for zero-shot humanoid deployment.Source: §5 · p. 8

Physics Coverage: The Physical Diversity Gap

• Grounding long-tail physical diversity requires bespoke tuning of material properties and boundary conditions, whose engineering cost may exceed collecting real-world data.The examples span object damping, garment stiffness, oil-stain friction, and granular food mechanics.Source: §6 · p. 9

Task Coverage: The Long-Tail of Task Generation

• Even with perfect physics simulation, scaling task coverage remains difficult because tasks require modeling affordances, object states, and interaction logic.The passage also notes that real deployment can surface unknown task variants that simulation cannot anticipate.Source: §6 · p. 9

Reward and Policy Coverage: The Reward Engineering Bottleneck

• Reward design must balance exploration against specification gaming, and manually tuning robust rewards for thousands of tasks is intractable.The authors characterize sim-to-real as scalable data generation with high upfront engineering effort.Source: §6 · p. 9

Hardware Coverage: The Hardware-Simulation Gap

• Dexterous hardware may have unmodeled friction, backlash, thermal throttling, and sensor noise, limiting transfer of policies that depend on precise finger positioning or force feedback.The passage contrasts idealized simulated actuation with current humanoid hardware.Source: §6 · p. 9

Outlook

• The authors judge general-purpose loco-manipulation through sim-to-real likely out of reach in the near future, while proposing simulation as a complement to real-world imitation learning and foundation models.They say the manipulation equivalent of locomotion’s sim-to-real “sweet spot” remains undiscovered.Source: §6 · p. 9

7.1. Observation Details

• Student input combines state observations with a 128-dimensional visual feature extracted from a 108 × 192 RGB image.Tables 1 and 2 summarize teacher and student observation terms and dimensions, respectively.Source: §7.1 · p. 14, Table 2 · p. 14, Table 1 · p. 14

7.2. Reward Details

• A place–pickup cycle has five stages, and its per-step reward is a stage-weighted sum with positive weights and stage-specific advancement and completion criteria.Table 3 summarizes reward components, expressions, weights, and the stages where each term applies.Source: §7.2 · p. 14, Table 3 · p. 15

7.3. Hyperparameters Details

• Tables 4–6 list the Unitree G1's joint-space PD gains, the teacher's PPO hyperparameters, and the student's DAgger–behavior-cloning hyperparameters.The student policy uses a mixture of DAgger and behavior cloning.Source: §7.3 · p. 14, Table 4 · p. 16, Table 5 · p. 16

7.4. Domain Randomization

• Student training randomizes image quality, dome lighting, materials, table properties, and camera extrinsics; Table 7 summarizes the randomization parameters.Source: §7.4 · p. 14, Table 7 · p. 17