Source-linked AI summary
HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesis
Lingwei Dang, Juntong Li, Zonghan Li, Hongwen Zhang, Liang An, Wei Min, Yebin Liu, Qingyao Wu
TL;DR
Multi-view hand-object interaction synthesis must handle fine-grained motion, occlusions, and coherent 3D geometry. HarmoHOI jointly diffuses appearance and 3D motion with global alignment and curriculum learning, achieving high visual quality, motion plausibility, and cross-view consistency.
Problem
Synchronized multi-view HOI generation with coherent 3D geometry and motion remains under-explored despite video foundation models’ visual priors.
Method
HarmoHOI jointly models multi-view appearance and 3D motion using M2DiT, GloMAD, and hybrid curriculum training in a closed-loop diffusion framework.
Results
HarmoHOI achieves high visual quality, motion plausibility, and cross-view consistency, with the best multi-view consistency in comparisons.
Takeaways & Limitations
The framework provides synchronized multi-view HOI video and motion co-generation that remains effective under hand-object occlusions.
Takeaways & Limitations
The available TACO data cover only 12 viewpoints, limiting denser multi-view training and arbitrary-view rendering.
Abstract
from arXiv · showhide
Hand-Object Interaction (HOI) synthesis is a cornerstone for animation production and embodied AI. Despite the strong priors of video foundation models, multi-view consistent HOI synthesis remains challenging due to complex hand motions and occlusions. We present HarmoHOI, a unified diffusion framework that jointly and harmoniously generates synchronized multi-view HOI videos and globally aligned 3D point tracks. Our core insight is that robust multi-view consistency fundamentally requires globally aligned 3D geometry and motion. To this end, we propose a Mixture of Multi-view Diffusion Transformer that co-models RGB videos and 3D point tracks. By representing point tracks as pseudo-videos, we align 3D geometric signals with the 2D latent space of foundation models, thereby minimizing the domain gap and easing adaptation of priors. To further ensure geometry consistency, we introduce Global Motion Aligning Diffusion, which refines coarse point tracks into metric-scale, globally aligned 3D trajectories. HarmoHOI enables on-the-fly co-evolution of 2D appearance and 3D motion during denoising. To overcome the scarcity of multi-view HOI data, we employ a hybrid data curriculum learning strategy that successfully transfers generic priors from single-view data to synchronized multi-view generation. Experimental results show that HarmoHOI achieves state-of-the-art performance in visual quality, motion plausibility, and multi-view geometric consistency. Project page available at https://droliven.github.io/HarmoHOI_project.
1 Introduction
HarmoHOI addresses the under-explored challenge of generating synchronized multi-view HOI videos with coherent 3D geometry and motion. It jointly harmonizes appearance and globally aligned 3D motion through diffusion modeling, trajectory refinement, and hybrid curriculum learning.
- Motivation and challenge: HOI synthesis supports animation and embodied dexterous manipulation but is difficult because of fine-grained hand movements, severe occlusions, and complex local deformations.Video foundation models provide strong visual priors, yet synchronized multi-view HOI generation with coherent 3D geometry and motion remains under-explored.
- Core framework: HarmoHOI proposes a joint diffusion framework that harmonizes multi-view HOI appearance with globally aligned 3D motion.Its central premise is that synchronized multi-view consistency depends on globally aligned 3D geometry and motion rather than treating 3D signals as external controls.
- Core framework: M2DiT jointly models 2D videos and 3D motion using camera conditioning, intra-view spatio-temporal modeling, inter-view geometric attention, and bidirectional mutual modulation.Point tracks are converted into normalized, color-mapped motion pseudo videos so a motion VAE can encode them into a latent space aligned with RGB videos.
- Core framework: GloMAD refines coarse, up-to-scale multi-view point tracks into globally aligned metric-scale 3D motions through scale regression and inter-view geometric attention.Sharing the diffusion pipeline with M2DiT enables an on-the-fly closed-loop feedback mechanism for mutual enhancement.
- Training strategy: A hybrid progressive curriculum first trains on single-view HOI videos with pseudo-geometric annotations, then gradually introduces synchronized multi-view videos and geometric data.This strategy transfers generic appearance and motion priors while learning multi-view epipolar geometric consistency despite scarce synchronized multi-view HOI data.
2 Related Work
Prior work on multi-view synthesis spans novel-view rendering, monocular-to-multi-view reconstruction, and direct synchronized generation, while HOI methods separately advance video synthesis and 3D motion generation. These approaches remain limited by consistency, ill-posed monocular reconstruction, temporal artifacts, pose-sequence requirements, or reliance on high-precision motion-capture data.
- Novel-view and Multi-view Video Synthesis: Multi-view video synthesis follows three paradigms: camera-controlled novel-view rendering, monocular-input multi-view reconstruction, and direct synchronized multi-view generation.The first paradigm has received the most attention, with data-driven novel-view methods and reconstruction-warping-inpainting pipelines using explicit 3D representations.
- Novel-view and Multi-view Video Synthesis: Single-trajectory methods cannot guarantee multi-view consistency, while monocular-input reconstruction is ill-posed because it infers multi-view 3D information from a single video.Examples include ReCamMaster and MV-Custom for novel-view generation, and SV4D 2.0 and MV-Performer for monocular-to-multi-view reconstruction.
- HOI Video Models: Pose-guided HOI video methods extend UNets with pose and appearance controls, but temporal modeling can cause flickering and requires predefined pose sequences.More recent works instead leverage diffusion transformers, as described in the supplied passage.
- 3D HOI Generation: 3D HOI generation primarily relies on high-precision 3D motion-capture data, with some methods predicting intermediate contact maps or affordances to improve kinematic plausibility.The cited works span motion capture and intermediate-structure prediction approaches.
3 Method
HarmoHOI synthesizes synchronized multi-view HOI videos together with metric-scale 3D point tracks from a reference image, camera poses, and a text prompt. Its pipeline combines pseudo-video motion representation, dual-branch diffusion modeling, global trajectory refinement, and progressive hybrid-data training.
- Problem Formulation: Given a reference image, target camera poses, and a textual prompt, HarmoHOI generates synchronized multi-view HOI videos and metric-scale 3D point tracks.The outputs span V viewpoints, T frames, image dimensions H×W, and K 3D points.
- Framework Overview: The framework uses M2DiT to jointly generate multi-view RGB videos, motion pseudo videos, and global metric scales, followed by GloMAD refinement of coarse tracks into globally synchronized 3D trajectories.GloMAD treats coarse point tracks as conditioning for conditional generation because pixel-wise optimization can produce inaccurate trajectories.
- Data Representation and Embedding: HarmoHOI converts 3D point tracks into pseudo videos and reuses a pretrained video foundation model to reduce the 2D–3D domain gap during motion generation.This avoids training a 3D point-track generator from scratch and mitigates collapse associated with directly supervised fine-tuning of video models.
- Mixture of Multi-view Diffusion Blocks: M2DiT extends single-view video DiT blocks with inter-view geometric attention and bidirectional cross-branch modulation for multi-view correspondence and 2D–3D consistency.Each block sequentially applies intra-view spatiotemporal attention, inter-view geometric attention, cross-branch modulation, text-conditioned cross-attention, and an MLP.
- Hybrid-data Progressive Curriculum: A hybrid-data progressive curriculum transfers visual priors from in-the-wild single-view HOI videos while progressively increasing geometric fidelity and multi-view consistency.The strategy exploits complementary laboratory datasets with accurate 3D dynamics and large-scale in-the-wild videos with diverse visual interaction scenarios.
4 Experiments
Experiments evaluate HarmoHOI on synchronized multi-view video and 3D motion generation using dedicated quality, consistency, and motion metrics. Results show strong multi-view video and 3D motion performance, while ablations attribute gains to simultaneous multi-view generation, joint video-motion diffusion, and motion alignment.
- Metrics: The evaluation measures single-view fidelity, dynamic quality, pixel alignment, and multi-view semantic consistency for generated videos.Subject Consistency and Dynamic Degree assess temporal stability and dynamics, while Matching Pixels and CLIP-Views evaluate multi-view consistency.
- Qualitative and Quantitative Comparison: HarmoHOI achieves the best multi-view consistency in video generation, while WAN 2.1 shows severe distortion and poor plausibility.The comparison uses multiple metrics and visual demonstrations because any single metric provides only a partial view.
- Qualitative and Quantitative Comparison: HarmoHOI demonstrates superior quantitative and qualitative 3D motion performance, unlike baselines with obvious layer-wise multi-view offsets.Its gains stem from jointly generating multi-view consistent points rather than reconstructing 3D motion independently per viewpoint.
- Ablation Study: Sequential viewpoint generation severely degrades multi-view consistency and 3D motion quality when HarmoHOI is adapted to single-view generation.The failure arises because sequential viewpoint generation cannot maintain consistent HOI patterns.
- Ablation Study: Removing video-motion joint diffusion improves consistency over single-view generation but still underperforms HarmoHOI, producing blurriness and deformation.Excluding GloMAD further reduces motion quality because the video foundation model lacks 3D motion training and VAE processing loses information.
5 Conclusion
The method co-generates synchronized multi-view HOI videos and motion while aligning global motion to improve visual realism, motion plausibility, and cross-view geometric consistency. Its main limitation is the scarcity of dense paired multi-view 2D and 3D motion data, restricting TACO to 12 viewpoints.
- Conclusion: The framework combines joint appearance-motion diffusion with global motion aligning diffusion for synchronized multi-view HOI video and motion co-generation.This combination targets visual realism, plausible motion, and cross-view geometric consistency.
- Conclusion: A multi-stage hybrid-data curriculum learning strategy enhances generalization, while requiring only a reference image and text instruction.The approach is described as particularly effective under hand-object interaction occlusions.
- Limitation: TACO is limited to 12 viewpoints because paired multi-view 2D videos and 3D motion data remain scarce.Collecting or synthesizing denser multi-view HOI data for a 4D Gaussian representation and arbitrary-view rendering is identified as future work.
—Supplementary Materials— · A Discussion on Related Works
The supplementary discussion examines limitations of leading video models for fine-grained hand-object interaction and situates HarmoHOI among representative multi-view generation methods through qualitative, quantitative, and schematic comparisons.
- A Discussion on Related Works: State-of-the-art video models still struggle with fine-grained hand-object interactions despite advances in visual quality and dynamic plausibility.The discussion tests Seedance 2.0 and WAN 2.7 as examples.
- A Discussion on Related Works: Seedance 2.0 hallucinates a hammer that should not exist, revealing weak instruction following.The issue appears in the reported HOI generation results.
- A Discussion on Related Works: WAN 2.7 produces unreasonable distortion of the wooden material in HOI generation.The figure marks hallucinations or distortion issues with red boxes.
- A Discussion on Related Works: Figure 10 compares HOI generation results from top-ranked video models and highlights hallucinations or distortion issues.The appendix also provides a video demonstration for a more vivid comparison.
- A Discussion on Related Works: Section 4 provides qualitative and quantitative comparisons with representative methods from each multi-view generation category.The supplementary materials use these comparisons to contextualize the reported generation results.
- A Discussion on Related Works: Table 4 presents schematic sketches and summarizes related methods’ properties in terms of generative capacity and 3D modeling.The passage introduces a broader property comparison beyond the visual examples.
B Close-loop Mutual Enhancement Cycle
HarmoHOI forms a closed-loop inference cycle between M2DiT and GloMAD: coarse motion guides point refinement, whose globally aligned results feed subsequent video denoising. Together, the modules co-generate multi-view videos and aligned 3D motion.
- Core components: M2DiT generates multi-view 2D videos, motion pseudo videos, and global metric scale, while GloMAD aligns per-view coarse point tracks into consistent 3D trajectories.These are the framework’s two core components.
- Closed-loop feedback: During inference, M2DiT outputs provide coarse motion conditions for GloMAD, while GloMAD points are projected and normalized into motion pseudo videos guiding M2DiT’s next denoising step.The process creates mutually enhancing feedback between appearance generation and 3D motion refinement.
- Inference procedure: The inference procedure initializes motion pseudo videos and 3D motion with Gaussian noise, then iteratively performs M2DiT co-generation and GloMAD conditional point refinement.Algorithm 2 outputs a multi-view HOI video and 3D motion from a text prompt, reference image, camera poses, VAE, GloMAD, and average depth scale.
C Additional Implementation Details.
HarmoHOI is implemented on the pretrained WAN 2.1-1.3B-T2V model with a compact video configuration and explicit tokenization for inter-view geometry. Its Global Motion Aligning Diffusion module uses Point Transformer V3 with sparse convolution.
- Backbone and resolution: HarmoHOI builds on WAN 2.1-1.3B-T2V, comprising 30 DiT modules with 12 attention heads each and total hidden dimension 1536.Each attention head has hidden dimension 128, and the maximum text token length is 512.
- Backbone and resolution: The model uses video resolution 49 × 256 × 384, while WAN 2.1’s VAE applies spatiotemporal compression ratios of (8 × 8 × 4).The visual tokenizer additionally performs camera-token and video-token processing.
- Tokenization and motion alignment: Camera, scale, and video tokens produce 3456 tokens for the inter-view Geometric Attention module, compared with 5760 tokens for the broader token sequence.The 5760-token sequence is computed as (2 + (⌊49/4⌋+ 1)) × (256/16) × (384/16), while the geometric-attention sequence is (1 + 2 + 6) × (256/16) × (384/16).
- Tokenization and motion alignment: GloMAD is built on Point Transformer V3 with sparse convolution.The supplied implementation details state that WAN 2.1 is first warmed up during training.
D Additional Demonstrations
The supplementary video provides diverse generated examples for image-to-video and motion generation, demonstrating varied interactions, objects, motions, and viewpoints. It also includes systematic visual comparisons with representative baselines to assess appearance quality.
- The method synthesizes temporally coherent videos with corresponding motion representations from a reference image.
- The supplementary video presents generated examples spanning different interaction scenarios, object categories, motion patterns, and camera viewpoints.
- Systematic visual comparisons with representative baseline approaches enable assessment of generation quality from appearance.