PaperlayerView exact revision on arXiv

Source-linked AI summary

GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors

Tianyi Xie, Haotian Zhang, Jinhyung Park, Zi Wang, Bowen Wen, Jiefeng Li, Xueting Li, Qingwei Ben, Haoyang Weng, Yufei Ye, David Minor, Tingwu Wang, Chenfanfu Jiang, Sanja Fidler, Jan Kautz, Linxi Fan, Yuke Zhu, Zhengyi Luo, Umar Iqbal, Ye Yuan

arXiv:2606.05160v1cs.RO

TL;DR

Scaling humanoid loco-manipulation demonstrations is difficult because teleoperation, motion capture, and physical scene setup are hard to scale. GRAIL generates robot-compatible data digitally from specified 3D scenes and video priors, achieving 84% pick-up success and 90% stair-climbing success on a Unitree G1.

  • Problem

    Scaling robot-compatible humanoid loco-manipulation demonstrations across diverse objects, motions, and terrains remains difficult with teleoperation, motion capture, and physical scene reconfiguration.

  • Method

    GRAIL specifies robot-centric 3D scenes, uses video foundation models to synthesize human interactions, reconstructs metric 4D trajectories, and retargets them to humanoid control.

  • Results

    84% pick-up success and 90% stair-climbing success were achieved in the real world using policies trained only on over 20,000 GRAIL-generated sequences.

  • Takeaways & Limitations

    GRAIL demonstrates that fully digital, robot-centric data generation can support sim-to-real deployment of task-general humanoid loco-manipulation policies.

  • Takeaways & Limitations

    GRAIL assumes suitable 3D assets, simulator-ready scenes, and a video foundation model that follows the requested interaction, while severe occlusion or motion can degrade reconstruction.

Abstract

from arXiv · show

Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We present GRAIL, a digital generation pipeline that remains fully virtual until deployment: it composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot. Rather than reconstructing unconstrained in-the-wild videos, GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction. This privileged setup better conditions 4D recovery, allowing model-based object tracking, human motion estimation, and interaction-aware optimization to reconstruct metric 4D human-object interaction (HOI) trajectories with reduced depth ambiguity and morphology mismatch. We retarget the recovered motions to a humanoid robot and train complementary task-general trackers: an object-aware latent adaptor for manipulation and a scene-aware tracker for terrain traversal. GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, we train egocentric visual policies through a sim-to-real pipeline and deploy them on a Unitree G1 humanoid, achieving 84\% real-world success on diverse object pick-up and 90\% success on stair-climbing.

1. Introduction

GRAIL is a fully digital, humanoid-centric pipeline that uses video foundation models within specified 3D assets and metric scenes to generate robot-compatible loco-manipulation data. It produces over 20,000 sequences and validates generated-data-trained visual policies on a Unitree G1, achieving 84% pick-up and 90% stair-climbing success.

  • Motivation: Humanoid loco-manipulation demonstrations must coordinate whole-body balance, object contact, and scene-aware locomotion across diverse objects and terrain geometries.Scaling these demonstrations is difficult because trajectories must be physically plausible and executable by the target robot.
  • GRAIL pipeline: GRAIL remains fully digital until deployment, using video foundation models as interaction priors inside a simulator-ready 3D asset pipeline.It specifies object geometry, scene geometry, camera parameters, metric scale, and a robot-proportioned character before recovering interactions.
  • Robot control: Recovered metric 4D human-object interaction trajectories are retargeted to a Unitree G1 and used to train complementary task-general tracking policies.The policies augment a pretrained whole-body controller for manipulation and terrain-conditioned whole-body control.
  • Generated dataset: Over 20,000 humanoid loco-manipulation sequences span pick-up, whole-body manipulation, sitting, and terrain traversal.The dataset is generated through GRAIL’s fully digital framework and is intended to provide physically plausible sequences.
  • Real-world validation: 84% pick-up success and 90% stair-climbing success are achieved in the real world by egocentric visual policies deployed on a Unitree G1.The policies are trained using only GRAIL-generated data through a sim-to-real pipeline with visual domain randomization and camera alignment.

2. Related Work

Prior humanoid supervision and human-object interaction data rely on motion capture, RGB-based reconstruction, and increasingly, human-video mining, retargeting, and robotized-video generation. The paper situates its approach within these directions while using asset-conditioned 4D HOI generation from specified 3D scenes.

  • Human-Object Interaction Generation and Reconstruction: Existing human-object interaction data sources use motion capture or RGB-based reconstruction to model human motion, object affordances, and physical contact.The cited motion-capture and RGB-reconstruction approaches form the prior data-generation landscape.
  • Human Video as Humanoid Supervision: Human video supervision increasingly supports humanoids through large-scale mining, motion retargeting, and robotized-video generation for pose control or pretraining.Related systems also train interaction policies from third-person, monocular, or egocentric video.
  • Asset-Conditioned 4D HOI Generation: Asset-conditioned 4D HOI generation uses specified 3D scenes and a robot-proportioned character to synthesize interactions and reconstruct metric human-object motion.The pipeline conditions video synthesis on rendered frames and jointly refines human and object trajectories using keypoint, depth, and contact losses.

3. Method

GRAIL generates humanoid loco-manipulation demonstrations through a fully virtual, three-stage pipeline: it synthesizes human interaction videos from known 3D configurations, reconstructs metric 4D trajectories, and retargets them into robot tracking data. The method uses known scene and camera geometry, interaction-aware optimization, and robot-proportioned characters to support physically coherent Unitree G1 deployment.

  • 3.1 Video Generation: GRAIL first assembles a robot-proportioned 3D configuration, renders an initial frame, and uses a VLM and VFM to synthesize a static-camera reference HOI video.The configuration includes known camera intrinsics and extrinsics; its environment supplies realistic visual context and ground-truth depth for reconstruction.
  • 3.3 Retargeting and Tracking: GMR retargets the robot-proportioned motion to Unitree G1, after which SONIC-based task-general tracking policies convert retargeted 4D HOI trajectories into robot-action data.Policies are trained across task families rather than separately per sequence or object, with related trajectories providing initialization for fine-tuning.
  • 3.2 4D HOI Reconstruction: Known object geometry, texture, camera parameters, and first-frame pose enable RGB-only 6-DoF object tracking, while fixed prefitted morphology reduces human reconstruction mismatch.GENMO estimates per-frame SMPL-X pose, WiLoR refines hand motion, and FoundationPose is fine-tuned for 5 epochs with depth channels zeroed.

4. Results

GRAIL generates physically executable 4D human-object interactions, scales them into task-general loco-manipulation data, and transfers egocentric visual policies to the Unitree G1. The resulting policies achieve 90% stair-climbing success and 84% object pick-up success in the real world.

  • HOI Generation: GRAIL achieves the strongest performance across nearly all HOI metrics, including contact distance, penetration ratio, interaction score, object smoothness, tracking success rate, body deviation, and object deviation.The evaluation uses 20 everyday objects and compares against CHOIS, HOIDiff, and DAViD.
  • Scaling Loco-Manipulation Data: Over 20,000 sequences span pick-up, whole-body manipulation, sitting, and terrain traversal across 1,000 object assets and 1,000 procedurally generated terrain configurations.Pick-up includes varied grasp strategies; manipulation includes carrying, pushing, and repositioning objects; terrain includes curbs, slopes, and stairs.
  • Task-General Loco-Manipulation Tracking: Removing SONIC or the object-aware latent adaptor substantially reduces manipulation success, showing that accurate body imitation alone is insufficient for object interaction.Vanilla SONIC has the lowest manipulation success rate despite the best body tracking.
  • Real-World Deployment: 90% real-world success is achieved for stair-climbing, while 84% real-world success is achieved for diverse object pick-up on a Unitree G1.The pick-up policy is trained on 200 approach-and-pick-up sequences per object across cubes, apples, tea boxes, carrots, and wet wipes.

5. Conclusion

GRAIL is a fully digital pipeline that generates humanoid loco-manipulation data from 3D assets and video priors, requiring the physical robot and environment only at deployment. Its fully specified 3D scenes control key reconstruction ambiguities, enabling robot-compatible trajectory recovery and sim-to-real deployment on a Unitree G1.

  • Conclusion: GRAIL generates humanoid loco-manipulation data digitally from 3D assets and video priors, using the physical robot and environment only at deployment.The pipeline is fully digital until deployment.
  • Conclusion: Egocentric visual policies trained only on GRAIL-generated data transfer to a Unitree G1 for object pick-up and stair-climbing.This demonstrates the pipeline’s sim-to-real deployment targets.
  • Conclusion: Fully specified 3D scenes provide object geometry and texture, camera parameters, metric scale, environment depth, and robot-proportioned morphology as controlled reconstruction inputs.These inputs replace uncontrolled-video reconstruction conditions and reduce key 4D HOI ambiguities.
  • Conclusion: The controlled setup enables model-based object tracking, metric depth alignment, and interaction-aware optimization to recover robot-compatible trajectories, which are retargeted to the Unitree.The passage identifies these mechanisms as enabling the recovered trajectories and retargeting process.

6. Limitations … A.3. Failure Case Filtering

GRAIL’s virtual generation pipeline depends on specified 3D assets, simulator-ready scenes, and compliant video foundation models, with reconstruction and policy-transfer limitations under difficult visual or motion conditions. Its generation process takes approximately 14 minutes per sequence, while automated mask-based filtering removes failures caused by tracking inconsistencies, fast motion, blur, or appearance changes.

  • 6. Limitations: GRAIL assumes 3D object assets, simulator-ready scene setups, and a video foundation model that follows the requested interaction.
  • 6. Limitations: Reconstruction quality degrades under severe occlusion, fast motion, or inconsistent object appearance from the video foundation model.
  • 6. Limitations: Task-general tracking policies still require training or fine-tuning when the motion family changes substantially.
  • A. Human-Object Interaction Generation Details: The pipeline constructs indoor floor-only or furnished-room scenes, places objects according to typical affordances, and pairs them with a human asset fitted to Unitree G1 morphology.
  • A.2. Generation Runtime: The video-generation and initial human-and-object motion-estimation stages together require about 4 minutes per sequence.
  • A.2. Generation Runtime: 14 minutes: the full 4D HOI generation pipeline takes approximately 14 minutes per 5-second, 121-frame sequence on a single NVIDIA A100 GPU.
  • A.2. Generation Runtime: 8 minutes: joint optimization dominates runtime by optimizing human and object trajectories across all frames.
  • A.3. Failure Case Filtering: The failure filter compares SAM2 object masks with predicted-pose masks and discards sequences when mask tracking error exceeds τ = 0.2.This targets artifacts such as texture inconsistencies or geometry mismatches that can cause FoundationPose to lose tracking.

B. Task-General Loco-Manipulation Tracking Details

GRAIL trains two physics-based tracking policies on SONIC: an object-aware adaptor for manipulation trajectories and a scene-aware tracker for terrain- and chair-conditioned trajectories. Their observations and reward terms are summarized in Tables 5 and 6.

  • Tracking policies: Two physics-based trackers build on SONIC, a pretrained whole-body controller.The trackers comprise an object-aware adaptor and a scene-aware tracker.
  • Tracking policies: The object-aware adaptor tracks manipulation 4D HOI trajectories, while the scene-aware tracker handles terrain- and chair-conditioned 4D HOI trajectories.
  • Tracking policies: Per-tracker policy observations appear in Table 5, and reward terms appear in Table 6.
  • Scene configurations: Two pre-built 3D scene templates support object placement: an indoor floor-only environment and a furnished room with a table.A VLM selects between them according to each object’s typical affordances.

B.1. Object-Aware Adaptor

The object-aware adaptor is a compact MLP that adds a learned latent residual and hand primitives to a frozen controller before decoding body-joint targets. It is trained with PPO in large-scale Isaac Lab simulation while the pretrained controller components remain frozen.

  • Architecture: The adaptor is a 3-layer SiLU MLP with hidden dimensions 512, 256, and 128, outputting a 66-dimensional meta-action.The meta-action contains 64 latent-residual dimensions and 2 left/right hand-primitive dimensions.
  • Architecture: The latent residual is scaled by λ = 0.1, added before finite scalar quantization, and decoded into 29 body-joint position targets.Hand primitives pass through sigmoid thresholding before execution.
  • Training: 30,000 PPO iterations train only the adaptor on 64 NVIDIA L40 GPUs with 1,024 parallel Isaac Lab environments per GPU.The pretrained encoder, FSQ quantizer, and decoder remain frozen throughout training.
  • Training: Reference motions come from a shared task-family 4D HOI pool, with reference-state initialization applied at every episode reset.The start frame is sampled uniformly from the first 30 frames and clipped before labeled hand-object contact.

B.2. Scene-Aware Tracker

The scene-aware tracker handles terrain traversal and sitting by replacing the flat-ground prior with end-to-end controller fine-tuning conditioned on reconstructed scene geometry. It uses height-map and auxiliary kinematic reconstruction signals to track motions across scene-level interactions.

  • Scene-Aware Tracker: Scene-level tasks use end-to-end controller fine-tuning with a height-map encoder and auxiliary kinematic decoder instead of a latent adaptor.The tracker targets stepping over curbs, traversing slopes and stairs, and sitting on chairs.
  • Architecture: The tracker represents terrain as an 11 × 11 height-map grid centered on the robot, spanning 1.5 m at 0.15 m resolution.Downward rays query terrain hits on the scene mesh, which are transformed into the robot’s yaw-aligned local frame.
  • Training: Training uses PPO for 30,000 iterations with 1,024 parallel environments per GPU on 64 NVIDIA L40 GPUs in Isaac Lab.The controller, height-map encoder, and parallel kinematic decoder are fine-tuned end-to-end.
  • Training: The kinematic decoder reconstructs motion targets with an auxiliary MSE loss weighted 0.01 to regularize the latent toward the reference motion.This reconstruction objective is trained jointly with the controller, encoder, and FSQ quantizer.
  • Training: Manipulation-specific grasp and object-pose tracking rewards are disabled because these scene-level tasks do not involve hand-object interaction.Episodes terminate under cumulative tracking-error thresholds with adaptive strict orientation and foot xyz constraints.

B.3. Training Cost

GRAIL amortizes tracking-policy training across shared task-family motion pools rather than fitting a separate policy per sequence. A full policy requires roughly 30 hours on 64 NVIDIA L40 GPUs and trains 2,000–4,000 motions jointly, yielding an amortized cost of about 0.5–0.9 minutes per motion.

  • Training Cost: Tracking policies are trained jointly over shared task-family pools instead of separately for each sequence.The reported training cost is therefore amortized per motion by dividing total run wall-clock time by the number of motions in the pool.
  • Training Cost: 30 hours on 64 NVIDIA L40 GPUs trains a full policy for 30,000 PPO iterations with 1,024 environments per GPU.Each run jointly trains 2,000–4,000 motions.
  • Training Cost: 0.5–0.9 minutes per motion is the reported amortized training cost across each 2,000–4,000-motion pool.This figure follows from dividing the roughly 30-hour training wall-clock by the motions trained jointly in a run.

C. Experiment Details · C.1. Human-Object Interaction Generation

GRAIL is evaluated against training-based and training-free 4D HOI generation methods, with user-study and qualitative results favoring its affordances and motion realism. Ablations show that interaction-aware reconstruction losses improve metric interaction quality and downstream humanoid tracking.

  • C.1. Human-Object Interaction Generation: GRAIL is compared with training-based CHOIS and HOIDiff, alongside training-free DAViD for 4D HOI generation.CHOIS uses language and sparse waypoints, HOIDiff uses affordance-conditioned diffusion, and DAViD uses image generation for the first frame.
  • C.1. Human-Object Interaction Generation: 30 participants preferred GRAIL by a wide margin for both appropriate object affordances and physically plausible motion.Each trial displayed sequences from three randomly selected methods among four.
  • C.1. Human-Object Interaction Generation: 75% is the theoretical upper bound for pick rates in the user study’s 3-of-4 random sampling protocol.The study used a shared 20-object evaluation set and measured affordance realism and physical realism.
  • C.1. Human-Object Interaction Generation: GRAIL produces more coherent motions, accurate contact, and natural hand poses than baselines on the shared evaluation set.Baseline methods often produce unrealistically flat or static hand configurations unsuitable for downstream humanoid skill learning.
  • C.1. Human-Object Interaction Generation: Removing L_proj degrades image-space consistency, whereas removing L_depth or L_cont weakens metric interaction quality on the 124-motion benchmark.The ablation targets interaction-aware reconstruction losses.
  • C.1. Human-Object Interaction Generation: Each reconstruction-loss ablation substantially reduces tracking success rate and increases trajectory deviation downstream.Reconstruction errors propagate into humanoid tracking quality, including object-position and trajectory metrics.
  • C.1. Human-Object Interaction Generation: The full model achieves the best overall downstream tracking quality despite not minimizing every reconstruction proxy in isolation.Table 8 reports reconstruction and downstream tracking metrics for each loss ablation.
Loading 2606.05160v1…