Source-linked AI summary
Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatars
Youliang Zhang, Zhengguang Zhou, Zhentao Yu, Ziyao Huang, Teng Hu, Sen Liang, Guozhen Zhang, Ziqiao Peng, Shunkai Li, Yi Chen, Zixiang Zhou, Yuan Zhou, Qinglin Lu, Xiu Li
TL;DR
Grounded human-object interaction for talking avatars remains challenging because avatars must perceive their environments and balance control with video quality. InteractAvatar decouples perception and planning from video synthesis through dual-stream modules that co-generate interaction motions and videos. On GroundedInter, it significantly outperforms existing approaches in producing plausible, controllable, and high-quality grounded interactions.
Problem
Existing talking-avatar methods do not adequately support text-aligned, environment-aware human-object interaction, while GHOI requires balancing controllability against video fidelity.
Method
InteractAvatar uses a dual-stream Diffusion Transformer with PIM for scene-aware text-aligned motion planning and AIM for audio-conditioned video synthesis with motion-to-video alignment.
Results
InteractAvatar significantly outperforms existing approaches on GroundedInter in generating plausible, controllable, and high-quality grounded human-object interaction videos.
Takeaways & Limitations
The framework provides text-driven grounded interaction generation for talking avatars and a benchmark for fine-grained GHOI evaluation.
Abstract
from arXiv · showhide
Generating talking avatars is a fundamental task in video generation. Although existing methods can generate full-body talking avatars with simple human motion, extending this task to grounded human-object interaction (GHOI) remains an open challenge, requiring the avatar to perform text-aligned interactions with surrounding objects. This challenge stems from the need for environmental perception and the control-quality dilemma in GHOI generation. To address this, we propose a novel dual-stream framework, InteractAvatar, which decouples perception and planning from video synthesis for grounded human-object interaction. Leveraging detection to enhance environmental perception, we introduce a Perception and Interaction Module (PIM) to generate text-aligned interaction motions. Additionally, an Audio-Interaction Aware Generation Module (AIM) is proposed to synthesize vivid talking avatars performing object interactions. With a specially designed motion-to-video aligner, PIM and AIM share a similar network structure and enable parallel co-generation of motions and plausible videos, effectively mitigating the control-quality dilemma. Finally, we establish a benchmark, GroundedInter, for evaluating GHOI video generation. Extensive experiments and comparisons demonstrate the effectiveness of our method in generating grounded human-object interactions for talking avatars. Project page: https://interactavatar.github.io
1. Introduction
Grounded human-object interaction extends talking avatars from facial and simple body animation to environment-aware, text-driven interaction. InteractAvatar addresses this through decoupled perception, planning, and video synthesis, evaluated on GroundedInter.
- 1. Introduction: Grounded human-object interaction requires avatars to perceive environments, act around designated objects, remain consistent with the initial frame, and follow text without explicit trajectories.The task also operates on scene objects without requiring additional object conditions.
- 1. Introduction: Existing audio-driven methods lack explicit object and environment modeling, while pose-driven methods require costly skeletal sequences that may not fit the reference context.These paradigms therefore expose complementary limitations in environmental awareness and user control.
- 1. Introduction: GHOI poses a scene-action grounding challenge and a control-quality dilemma between instruction following and video fidelity.Methods may produce high-fidelity videos with weak instruction adherence or reasonable responses with reduced fidelity.
- 1. Introduction: InteractAvatar uses a dual-stream Diffusion Transformer that decouples perception and planning from video synthesis.Its Perception and Interaction Module generates text-aligned motions, while the Audio-Interaction-aware Generation Module synthesizes videos conditioned on audio and motion features.
- 1. Introduction: GroundedInter contains 600 test cases pairing reference images, structured interaction descriptions, and speech audio for GHOI evaluation.Experiments on this benchmark show significant improvements over existing approaches in plausible, controllable, and high-quality interaction videos.
2. Related Work
Related work progresses from audio-driven talking avatars to human-object interaction and subject-consistent video generation. However, existing approaches either edit footage, require complex controls, or generate new scenes instead of grounding interactions in a supplied environment.
- 2. Related Work: Audio-driven human animation evolved from head-and-shoulder lip-sync systems toward half-body and full-body digital human synthesis with body action modeling.Examples improve realism, lip synchronization, stability, or natural motion through motion coefficients, hierarchical diffusion, facial landmarks, and temporal modules.
- 2. Related Work: Human-object interaction methods edit existing footage or replace humans and handheld objects, but cannot animate a static reference image from scratch.Other approaches pursue fine-grained control through complex conditional inputs.
- 2. Related Work: Subject-consistent video methods combine reference-image consistency with HOI or audio-driven animation, yet typically generate an entirely new scene rather than using the provided environment.This distinguishes them from environment-grounded interaction generation.
3. Method
InteractAvatar separates scene-aware motion planning from high-fidelity video rendering in two aligned DiT streams, while jointly supporting text, image, audio, and interaction-motion conditioning.
- Architecture: The dual-stream DiT decomposes grounded interaction generation into PIM for perceptual planning and AIM for video rendering.PIM generates scene-aware, text-aligned motions; AIM synthesizes high-fidelity videos with audio lip synchronization.
- Perception and Interaction Module: PIM parses a reference image and text prompt to generate spatiotemporally reasonable interaction motion representing human poses and object boxes.The motion representation combines skeletal keypoints with object trajectories or boxes.
- Perception and Interaction Module: Task embeddings distinguish pure action generation from HOI generation, which additionally models object trajectories alongside human pose sequences.The task embedding is concatenated with the text embedding for cross-attention conditioning.
- Perception and Interaction Module: Environment-perception training alternates motion continuation with perception-as-generation, sometimes reducing the target length to one frame for detection-like learning.The unified flow-matching objective trains shared parameters for scene parsing and temporal motion generation.
- Audio-Interaction Aware Generation Module: AIM extracts contextual Wav2Vec audio features and injects them through cross-attention, with a face mask concentrating the signal on facial regions.The contextual audio window captures co-articulation and temporal speech dynamics, while spatial weighting stabilizes training.
- Motion-to-Video Alignment: The M2V aligner injects upsampled, zero-initialized PIM residuals into corresponding AIM layers, enabling synchronized motion control while reducing training instability and skeletal ghosting.The aligner uses bilinear interpolation and a zero-initialized linear projection before residual injection.
- Multimodal Training: AIM training exposes the model to audio before interaction motion because the two modalities differ in softness, locality, and heterogeneity.The reported strategy prioritizes audio synchronization before adding interaction-motion conditioning.
4. Experimental Results and Analysis
Experiments on GroundedInter and real scenes show that InteractAvatar supports text-aligned, plausible interactions while preserving reference environments. Ablations attribute these results to environmental perception training, PIM–AIM joint guidance, RGB motion representation, and multimodal training design.
- Benchmark: GroundInter contains 400 annotated images with 1–3 objects, 100 common object categories, action descriptions, and matching dialogue scripts.The benchmark is constructed for evaluating grounded human-object interaction generation.
- Quantitative comparison: InteractAvatar’s TA2V variant improves approximately 180% in HQ and 111% in OQ over Wan-S2V while maintaining comparable DINOsubject and DINOref.It also outperforms audio-driven methods on lip synchronization in the HOI scenario.
- Quantitative comparison: Pose-driven UniAnimate-DiT underperforms the TAM2V variant across metrics because pose-driven methods do not account for object morphology, causing interaction misalignments and unnatural deformations.UniAnimate-DiT uses motion sequences generated by PIM as input, yet the TAM2V variant achieves a particularly notable margin on OQ.
- Qualitative evaluation: Qualitatively, InteractAvatar follows textual instructions and preserves stable, plausible interactions with objects in the reference-image environment.The PIM plans scene-aware interaction motions while the AIM synthesizes photorealistic interaction details.
- Ablations: Environmental perception training shows that prepending the reference image performs better than appending it when modified RoPE limits excessive reference-image influence.Detection data establishes environmental perception, while Det.+Cont. data bridges perception and generation.
- Ablations: Joint PIM–AIM training outperforms cascaded guidance because cascaded motion conditions create an information bottleneck and restrict modeling of object-shape changes.Layer-wise residual injection with a zero-initialized linear layer outperforms simple layer-wise addition and helps prevent skeletal-line ghosting.
- Ablations: PIM improves semantic interaction quality: frozen-PIM cascades produce object deformations, while models without PIM often execute requested interactions inaccurately.The proposed joint model generates reasonable, text-aligned interaction videos.
- Ablations: RGB motion representation improves text-to-motion accuracy and generalization over simpler 2D coordinates by sharing a representation space with video generation.The shared RGB space also simplifies video-motion alignment and reduces training complexity.
5. Conclusion
InteractAvatar generates grounded human-object interaction videos through a dual-stream talking-avatar framework that separates perception and planning from video synthesis. The paper also introduces a benchmark for evaluating GHOI generation, while noting that the method currently handles only single-person scenes.
- InteractAvatar decouples perception and planning from video synthesis to generate plausible, controllable, high-quality GHOI videos.PIM perceives the reference-image context and plans text-aligned interaction motion, while AIM generates talking and interacting avatars using an M2V aligner.
- The framework combines PIM for scene-aware, text-aligned interaction motion with AIM for vivid talking-avatar video generation.
- A benchmark is proposed to evaluate grounded human-object interaction generation.
- The method is limited to single-person scenes and cannot generate multi-person human-object interactions.