EMO2: End-Effector Guided Audio-Driven Avatar Video Generation

• Audio-driven video generation still struggles to produce vivid, synchronized body motion, while full- or half-body prediction faces complex multi-joint coupling.The correlation between audio and body joints varies significantly.Source: §1 · p. 1
• Hands act as end-effectors tied more directly to human intention and audio than other body joints, making them a more intuitive motion target.The paper motivates this with robotic end-effector planning, where the rest of the body cooperates with the end-effector through inverse kinematics.Source: §1 · pp. 1–2, Figure 1 · p. 1
• The paper’s “pixels prior IK” approach uses hand motion to represent partial body movement while relying on pixel generation to produce other body parts with implicit inverse-kinematics priors.The paper notes that inverse kinematics can yield singularities and suboptimal motion-planning solutions.Source: §1 · p. 2
• Stage one maps audio to hand poses, reducing mapping-space complexity and allowing more precise gesture control; stage two uses diffusion to synthesize frames incorporating those poses.Source: §1 · p. 2
• The framework is reported to generate coherent, continuous co-speech videos with synchronized lip movements and realistic body dynamics.Source: §1 · p. 2
• The method outperforms state-of-the-art approaches in visual quality, synchronization accuracy, and motion diversity.Source: §1 · p. 2

Co-speech gesture generation.

• Data-driven co-speech gesture methods learn speech-motion connections with neural architectures, including transformers, codebooks, autoencoders, VQ-VAEs, and GAN pipelines.Source: §2 · p. 2
• Diffusion gesture methods use skeleton or SMPL pose sequences, or audio control networks, while few methods apply diffusion directly to hand postures.The passage notes that predefined body structures can constrain final gesture quality.Source: §2 · p. 2

Audio-driven human video generation.

• Audio-driven human-video work has advanced, but much research focuses on talking heads; EMO targets expressive talking-head video, while Loopy and Hallo address sequence history and audio control.Source: §2 · p. 2
• Body-animation approaches include Vlogger’s 3D-human pipeline, Cyberhost’s pixel-space body driving, Echomimic’s partial-body controls, and latent motion signals; Vlogger is described as insufficiently correlated with audio.Source: §2 · p. 3

3.1. Overview

• The two-stage pipeline translates audio into partial-body motion signals, then combines audio and motion signals to animate a single reference image into co-speech video.The second stage uses a ReferenceNet-based diffusion architecture.Source: §3.1 · p. 3

3.2. Preliminaries

• Diffusion training learns to denoise noisy data under control signals; Stage 1 uses a transformer, while Stage 2 uses a ReferenceNet-based denoising U-Net in latent space.The Stage 2 VAE maps image features into a latent space to reduce computational load.Source: §3.2 · p. 3

3.3. Partial Body Motion Generation

• Hand motion alone serves as the Stage 1 driving signal, with a DiT generating speech-rhythmic MANO coefficients through audio cross-attention and self-attention over motion latents.The model concatenates the previous clip’s final motion frames for smoother transitions; masks mark valid annotations and sequence padding, while style and speed embeddings control motion.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

3.4. Co-Speech Video Generation

• Stage 2 denoises video latents using reference-image and motion-frame features, audio, and Stage 1 MANO and keypoint maps to guide identity, synchronization, and body motion.Hand-confidence embeddings condition hand rendering; temporally filtered, intentionally misaligned keypoints guide movement while preserving creative freedom, and a pose discriminator encourages body structure.Source: §3.4 · pp. 4–6, Figure 3 · p. 5

4.1. Implement Details

• The hand-motion model uses a MANO representation, while video-model training proceeds from image training to audio-video training with temporal and audio-attention modules.The video model uses 24-frame clips at 704 × 512 resolution; training data totals approximately 275 hours.Source: §4.1 · p. 6

4.2. Hand Motion Generation Comparisons

• On Talkshow hand-motion comparisons, the MANO method leads in DIV and BA, while its lower PCK and FGD reflect greater freedom to move differently from ground truth.The authors associate DIV with hand-motion vividness and expressiveness; their SMPL-based method also leads baselines in diversity.Source: §4.2 · p. 7

4.3. Video Generation Comparisons

• Video comparisons use EMTD and assess frame quality, temporal coherence, identity consistency, lip-audio synchronization, hand representation, motion richness, and expression divergence.CyberHost and Vlogger comparisons rely on homepage demos because their models were not open-sourced.Source: §4.3 · pp. 7–8
• Our method outperforms EchoMimicV2 and MimicMotion on video-quality assessment, with lower FVD and improved FID, SSIM, and PSNR scores.The “w/o motion gen” variant uses ground-truth pose sequences; the authors note that this naturally improves quality metrics, while higher HKV indicates more diverse motion.Source: Table 2 · p. 7, §4.3 · p. 8
• Against CyberHost and Vlogger demos, our model has the highest HKV, and Figure 6 shows higher-quality frames that better preserve facial features.The authors attribute Vlogger’s relatively high HKC to minimal motion, indicated by very low HKV.Source: §4.3 · p. 8

5. Conclusion

• The framework generates hand movements in Stage 1 and uses them as control signals for Stage 2 video generation.The paper calls its rationale “pixels prior IK”: hand movements are described as most correlated with audio, while the video backbone naturally incorporates human inverse-kinematics priors in pixel form.Source: §5 · p. 8
• The authors report that the framework produces more expressive and vivid human videos than other methods.Source: §5 · p. 8
• The paper presents this framework as a new thought for audio-driven talking-head animation.Source: §5 · p. 8