Source-linked AI summary
HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesis
Mingjin Chen, Junhao Chen, Zhaoxin Fan, Yujian Lee, Zichen Dang, Lili Wang, Yawen Cui, Lap-Pui Chau, Yi Wang
TL;DR
HVG-3D targets the limited spatial expressiveness of 2D controls for hand-object interaction video synthesis and their restricted use of synthetic 3D conditions. It introduces a 3D ControlNet and hybrid real–simulation pipeline that generate videos from a real image and 3D input. On TASTE-Rob, it achieves state-of-the-art spatial fidelity, temporal coherence, and controllability while integrating real and simulated data.
Problem
2D conditioning signals limit spatial expressiveness and leave 3D geometry, motion, and synthetic conditional data insufficiently utilized for hand-object video synthesis.
Method
HVG-3D combines a 3D ControlNet for point-cloud and tracking cues with a hybrid pipeline that constructs real and simulated input-condition pairs.
Results
HVG-3D achieves state-of-the-art spatial fidelity, temporal coherence, and controllability on the TASTE-Rob benchmark.
Takeaways & Limitations
The framework supports controllable hand-object video synthesis from a single real image and 3D conditions obtained from simulation or real data.
Takeaways & Limitations
The task assumes a single input image, a sequence of 3D point clouds, and an optional 3D tracking sequence as conditioning inputs.
Abstract
from arXiv · showhide
Recent methods have made notable progress in the visual quality of hand-object interaction video synthesis. However, most approaches rely on 2D control signals that lack spatial expressiveness and limit the utilization of synthetic 3D conditional data. To address these limitations, we propose HVG-3D, a unified framework for 3D-aware hand-object interaction (HOI) video synthesis conditioned on explicit 3D representations. Specifically, we develop a diffusion-based architecture augmented with a 3D ControlNet, which encodes geometric and motion cues from 3D inputs to enable explicit 3D reasoning during video synthesis. To achieve high-quality synthesis, HVG-3D is designed with two core components: (i) a 3D-aware HOI video generation diffusion architecture that encodes geometric and motion cues from 3D inputs for explicit 3D reasoning; and (ii) a hybrid pipeline for constructing input and condition signals, enabling flexible and precise control during both training and inference. During inference, given a single real image and a 3D control signal from either simulation or real data, HVG-3D generates high-fidelity, temporally consistent videos with precise spatial and temporal control. Experiments on the TASTE-Rob dataset demonstrate that HVG-3D achieves state-of-the-art spatial fidelity, temporal coherence, and controllability, while enabling effective utilization of both real and simulated data.
1. Introduction
HVG-3D addresses the limited spatial expressiveness of 2D controls by conditioning hand-object video synthesis on explicit 3D representations. Its diffusion architecture and hybrid data pipeline support controllable generation from real images paired with simulated or real 3D conditions, achieving state-of-the-art results on TASTE-Rob.
- Motivation: 2D controls such as trajectories, optical flow, bounding boxes, and masks provide limited spatial and temporal information for hand-object interaction synthesis.The resulting challenges include incomplete 3D understanding and high data cost.
- Motivation: 3D tracking videos offer richer motion guidance, but projecting them into 2D prevents full use of spatial structure and depth relations.The paper motivates intrinsic 3D conditioning to improve physical plausibility and support scalable simulator-based data generation.
- Approach: HVG-3D uses a dedicated 3D ControlNet to encode geometric and motion cues from point clouds or tracking sequences for explicit 3D reasoning.The encoded features are injected into a diffusion transformer through zero-initialized convolutional layers.
- Approach: A hybrid pipeline pairs real images with 3D conditions from simulation or real videos, enabling flexible and precise control during training and inference.This bridges real and simulated domains within the data construction process.
- Results: HVG-3D significantly outperforms state-of-the-art methods across multiple metrics on TASTE-Rob, with superior spatial fidelity, temporal coherence, and controllability.The reported evaluation also highlights effective integration of real and simulated data for scalable video generation.
2. Related Works
Related work spans controllable video generation, hand-object interaction synthesis, and 3D rendering. Existing approaches provide spatial, temporal, or geometric control, but predominantly use 2D representations or lack full scene context; HVG-3D instead directly conditions diffusion on 3D point clouds.
- Controllable Video Generation: Controllable video methods guide appearance, pose, or motion using masks, keypoints, optical flow, trajectories, and camera-motion cues.Recent methods combine spatial and temporal signals and extend control to interactive multi-person scenarios.
- Hand-Object Interaction Generation: 3D hand-object methods reconstruct or generate poses, while 2D methods synthesize hand-object images and videos.Many 3D approaches operate on isolated objects without full scene context, limiting real-world applicability.
- 3D Rendering: Traditional rendering offers precise geometric control but requires extensive manual asset preparation, while neural rendering learns implicit or explicit 3D representations from multiview images.Rendering-then-generation can lose information and struggle with complex spatial relationships under occlusion.
- 3D Rendering: HVG-3D directly incorporates 3D point clouds as conditioning signals so diffusion operates as a neural renderer while maintaining 3D structural information.This contrasts with pipelines that render visual content first and then generate from the rendered result.
3. Methods
HVG-3D formulates hand-object image-to-video generation from a single image, 3D point clouds, and optional tracking, then combines a 3D-aware diffusion architecture with a hybrid signal-construction pipeline. A ControlNet injects 3D cues into a frozen video diffusion backbone, while the pipeline recovers and aligns real and synthetic conditions for training and inference.
- 3.1. Overview: The task generates a video from one RGB image, a sequence of hand-object point clouds, and an optional 3D tracking sequence.The output is intended to be visually realistic, temporally coherent, and consistent with 3D spatial constraints.
- 3.1. Overview: HVG-3D has two components: a 3D-aware diffusion architecture for geometric and motion cues, and a hybrid pipeline integrating real and simulated data.Both components provide spatial and temporal control during training and inference.
- 3.2. 3D-aware HOI Diffusion Architecture: The architecture uses a pretrained image-to-video diffusion backbone and a dedicated 3D point cloud-guided ControlNet to address the spatial limitations of 2D conditioning.CogVideoX-5B-I2V provides the Transformer-based video diffusion backbone with 3D full attention.
- 3.2. 3D-aware HOI Diffusion Architecture: The backbone encodes the input image and noised video latents, iteratively denoises them with a Diffusion Transformer, and decodes the clean latent into the output video.The architecture is designed for temporal modeling and fine-grained control over video dynamics.
- 3.2. 3D-aware HOI Diffusion Architecture: A point-cloud encoder produces latent geometric features, which are aligned with tracking and video latents before entering the 3D Point Cloud ControlNet.The point-cloud representation contains T frames and L latent tokens with 768-dimensional features.
- 3.2. 3D-aware HOI Diffusion Architecture: The ControlNet replicates pretrained DiT blocks and injects its 3D structural and motion outputs into corresponding backbone layers through zero-initialized convolutions.This supplies 3D cues at every denoising step.
- 3.3. Hybrid Pipeline for Input and Condition Signal Construction: The hybrid pipeline constructs training signals from monocular egocentric videos and supports real or synthetic 3D conditioning during training and inference.It addresses missing mask and point-cloud annotations through segmentation, tracking, and per-frame 3D reconstruction.
- 3.3. Hybrid Pipeline for Input and Condition Signal Construction: Training emphasizes hand-object regions through a mask-weighted latent reconstruction objective while fine-tuning copied condition blocks and freezing the original denoising backbone.Samples include images, videos, masks, point clouds, and tracking sequences; training uses 49-frame clips resized to 720 × 480.
4. Experiment
Experiments on TASTE-Rob compare HVG-3D with general-purpose and specialized baselines using full-frame, hand–object-region, qualitative, flexibility, and ablation evaluations. HVG-3D achieves strong quantitative and qualitative interaction fidelity while supporting heterogeneous 3D conditioning sources.
- Experiment Setting: The evaluation uses 100 randomly selected test videos from TASTE-Rob, with 49-frame clips sampled for comparison.The test set covers five scene categories from the Single Hand subset; each evaluation video contributes one hand–object interaction clip.
- Baseline Comparisons: Full-frame evaluation reports image-quality and spatiotemporal metrics against CogVideoX, Wan 2.2, Kling, DaS, and InterDyn.The metrics include L1, PSNR, SSIM, LPIPS, CLIP Score, FID, CLIP-FID, and spatiotemporal measures.
- Quantitative Results: 13.8 FVD, 58.2 FID, 0.96 CLIP Score, and 0.40 GMSD-T are reported as HVG-3D’s best full-frame values for those metrics.Within the hand–object mask region, FVD decreases from 13.8 to 9.6 and C-FID from 14.6 to 13.1, with improvements also reported for L1, LPIPS, PSNR, and SSIM.
- Qualitative Results: Qualitatively, HVG-3D executes folding, object movement, and placement manipulations while preserving hand and object shapes.DaS and InterDyn show deformation or instability on harder motions, while CogVideoX, Wan2.2, Kling, and Sora2 struggle to reliably complete the specified manipulation.
- Conditioning Flexibility: HVG-3D accepts 3D point clouds, simulator outputs, reconstructed driving-video conditions, datasets, and edited Blender mesh sequences at inference.These sources provide a unified interface for heterogeneous 3D inputs, including synthetic hand–object interaction sequences rendered by simulators.
- Ablation Study: Ablations show that 3D point clouds, 3D tracking videos, and mask diffusion loss improve interaction quality, spatial accuracy, and training convergence.Removing tracking video particularly reduces contact-location and motion-trajectory accuracy, while removing mask diffusion loss shifts attention away from the interaction region.
5. Conclusion
HVG-3D combines a 3D ControlNet with a hybrid real–simulated data pipeline for 3D-conditioned hand–object interaction video generation. On TASTE-Rob, it achieves state-of-the-art spatial fidelity, temporal coherence, and controllability, with ablations confirming complementary component benefits.
- HVG-3D uses a 3D ControlNet to encode point-cloud and tracking cues into a video diffusion backbone.
- A hybrid pipeline bridges real and simulated domains for constructing inputs and condition signals.
- On TASTE-Rob, HVG-3D achieves state-of-the-art spatial fidelity, temporal coherence, and controllability.
- Ablation studies confirm the complementary benefits of each component.