Source-linked AI summary
CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation
Xiangyang Luo, Xiaozhe Xin, Tao Feng, Xu Guo, Meiguang Jin, Junfeng Ma
TL;DR
HOI video diffusion must preserve sensitive human regions and physically plausible hand–object contact, weaknesses that arise from RGB-centric supervision. CoInteract embeds spatially supervised experts and dual-stream interaction-structure training into DiT, with the auxiliary branch removed at inference; experiments report consistent gains in structural stability, logical consistency, and interaction realism.
Problem
RGB-centric diffusion models frequently produce unstable hands and faces and physically implausible hand–object contact because they lack explicit structural and interaction-geometry supervision.
Method
CoInteract combines a spatially supervised Human-Aware Mixture-of-Experts with dual-stream RGB–HOI co-generation in a shared DiT backbone, removing the auxiliary HOI branch at inference.
Results
CoInteract consistently outperforms existing methods in interaction plausibility and structural stability while maintaining a zero-overhead inference path.
Takeaways & Limitations
The framework integrates structural fidelity and physical interaction priors into HOI video generation without requiring an auxiliary branch during inference.
Abstract
from arXiv · showhide
Synthesizing human--object interaction (HOI) videos has broad practical value in e-commerce, digital advertising, and virtual marketing. However, current diffusion models, despite their photorealistic rendering capability, still frequently fail on (i) the structural stability of sensitive regions such as hands and faces and (ii) physically plausible contact (e.g., avoiding hand--object interpenetration). We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio. CoInteract introduces two complementary designs embedded into a Diffusion Transformer (DiT) backbone. First, we propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead. Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors. During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation. Experimental results demonstrate that CoInteract significantly outperforms existing methods in structural stability, logical consistency, and interaction realism.
1 Introduction
HOI video synthesis requires coordinated human motion, object manipulation, and physically plausible contact, but RGB-centric diffusion models often distort hands and faces or violate contact geometry. CoInteract addresses these problems with structure-aware generation and spatially specialized processing embedded in a DiT backbone.
- HOI synthesis extends beyond talking avatars by requiring coordinated hand movements, precise object manipulation, and strict physical plausibility.
- RGB-centric diffusion models lack explicit structural and spatial relationship awareness, causing hand and face collapse alongside hand–object interpenetration.
- CoInteract is an end-to-end framework that embeds human structural priors and HOI physical constraints directly into a Diffusion Transformer backbone.
- A Human-Aware Mixture-of-Experts routes tokens to lightweight hand- and face-specialized experts using bounding-box spatial supervision, adding only marginal parameters.
- A dual-stream paradigm jointly trains RGB and auxiliary HOI structure streams, using silhouette-based human structure and RGB objects to enforce physical plausibility.
- An asymmetric co-attention mask embeds physical interaction rules into DiT, reducing hand–object interpenetration with zero additional inference cost.
2 Related Works
Related work includes RGB-centric video diffusion, audio-driven human animation, dedicated hand and face refinement, and two HOI-generation paradigms. CoInteract combines interaction-structure supervision with backbone-integrated region specialization and removes auxiliary structure processing at inference.
- Video diffusion and animation: RGB-centric video diffusion models provide weak constraints on HOI contact geometry and body topology, producing hand or face distortions and contact violations.
- Video diffusion and animation: Audio-driven animation largely targets talking heads, avatars, or human-only motion and does not explicitly enforce hand–object contact constraints in rendered videos.
- Sensitive-region modeling: Existing hand and face methods commonly operate as external add-ons, whereas CoInteract embeds a Human-Aware MoE directly into DiT blocks.
- HOI video generation: Multi-condition HOI methods use pose and object controls but depend on heavy preprocessing and external signals that may limit robustness, generalization, or backbone-internalization of constraints.
- HOI video generation: Multi-reference methods provide identity and product references without explicit geometry conditions, often yielding unstable hand articulation and implausible contact.
- HOI video generation: CoInteract jointly trains RGB and auxiliary HOI structure streams with a Human-Aware MoE, then removes the auxiliary branch for zero-overhead inference.
3 Method
CoInteract jointly trains RGB appearance and texture-stripped HOI structure streams in a shared DiT, combining spatially structured co-generation with region-specialized routing. Its asymmetric attention transfers interaction supervision during training while allowing HOI removal at inference, and its Human-Aware MoE targets hand and face fidelity.
- Unified RGB–HOI Co-Generation: CoInteract jointly models RGB appearance and auxiliary HOI structure streams in shared DiT blocks to guide physically consistent representations.The HOI stream preserves interaction geometry while discarding appearance cues, and both streams use shared transformer parameters with stream-specific modulation.
- Unified RGB–HOI Co-Generation: The joint flow-matching objective supervises both RGB and HOI streams using L_flow = L_r + λ_h L_h.The conditioning includes text, audio, dual reference images, and motion latents, with λ_h set to 1 unless otherwise stated.
- Unified RGB–HOI Co-Generation: 3D RoPE assigns distinct spatial and temporal coordinates to RGB, HOI, motion, and reference tokens to encode cross-stream alignment and temporal relationships.RGB and HOI streams use separate width coordinates with shared height and time indices; historical motion receives negative time indices and references far-field temporal locations.
- Two-Stage Asymmetric Co-Attention: Two-stage asymmetric co-attention first couples the streams with full attention, then restricts RGB queries to RGB tokens while HOI queries attend to both streams.HOI loss backpropagates through HOI-to-RGB attention into shared DiT parameters, enabling the RGB pathway to retain the learned interaction supervision after HOI removal.
- Human-Aware Mixture-of-Experts: The Human-Aware MoE routes tokens through shared, head, hand, and base experts using spatial supervision for specialized hand and face processing.A stop-gradient router uses face and hand bounding boxes to assign region tokens to specialized experts, while remaining tokens use the base expert.
- Human-Aware Mixture-of-Experts: The framework improves hand sharpness and face identity consistency without degrading general synthesis quality.The HOI branch is removed at inference, preserving zero-overhead RGB generation after structure-aware training.
4 Experiments
CoInteract is evaluated against established baselines using automatic metrics, qualitative comparisons, a user study, and component ablations. Results show stronger interaction plausibility, hand and face fidelity, reference consistency, and efficient inference, while each proposed component addresses a distinct failure mode.
- Quantitative Comparison: CoInteract achieves the highest VLM-QA (0.72) and HQ (0.724), indicating strong interaction plausibility and hand structural stability.It also leads in DINOid (0.671), FaceSim (0.696), and Smooth (0.9951).
- Qualitative Results: Qualitatively, CoInteract maintains coherent hand articulation, natural product grasping, prompt adherence, and physically plausible interaction throughout sequences.Baselines show interpenetration, identity drift, unnatural grasping, or background deviation in different scenarios.
- Internal Mechanisms: The HOI stream preserves a synchronized geometric scaffold, while the router isolates face and hand tokens for specialized experts during complex interactions.Visualizations show alignment between RGB and HOI streams and routing heatmaps for face and hand experts.
- User Study: CoInteract achieves the lowest mean rank across Object Consistency, Human/Background Consistency, and Interaction Plausibility in the blind user study.The largest advantage is reported for Interaction Plausibility.
- Ablation Study: Removing MoE reduces HQ from 0.724 to 0.658 and FaceSim from 0.696 to 0.662, confirming its role in fine-grained structural fidelity.Qualitative ablations show hand collapse and blurred facial details without MoE.
- Ablation Study: Removing co-generation reduces VLM-QA from 0.72 to 0.48, while retaining the HOI branch raises inference cost to 4.13×.The MoE adds only 1.04× overhead relative to the MoE-free baseline, supporting the asymmetric strategy’s efficiency.
5 Conclusion
CoInteract is a structure-aware framework for speech-driven HOI video synthesis that combines spatially supervised Human-Aware MoE routing with asymmetric Spatially-Structured Co-Generation. The framework targets structural integrity and physical consistency while maintaining a zero-overhead inference path.
- Conclusion: CoInteract combines spatially supervised Human-Aware MoE routing with asymmetric co-generation to improve structural integrity and physical consistency in HOI videos.The HOI branch supplies interaction priors during training and is removed at inference.
- Conclusion: The framework reduces hand-object interpenetration and geometric misalignment while maintaining zero-overhead inference.The conclusion reports consistent gains in interaction plausibility and structural stability.