PaperlayerView exact revision on arXiv

Source-linked AI summary

VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis

Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, Cristian Sminchisescu

arXiv:2403.08764v1cs.CV

TL;DR

Generating realistic human videos from one image remains complex, while earlier work often targets lip synchronization alone. VLOGGER predicts stochastic 3D body-motion controls from speech and uses temporally and spatially controlled image diffusion; it reports outperforming prior state of the art on three public benchmarks.

  • Problem

    Realistic human video synthesis from a single image remains complex, and many earlier approaches focus exclusively on lip synchronization.

    Source: §1 · p. 2

  • Method

    VLOGGER predicts stochastic body-motion and facial-expression controls from audio, then uses a temporal image-to-image diffusion model conditioned on those controls and a reference image.

    Source: §1 · p. 3, §3 · p. 6

  • Results

    VLOGGER outperforms previous state-of-the-art methods in large quantitative comparisons on three public benchmarks and exhibits fairly low bias across evaluated diversity axes.

    Source: §1 · p. 4, §4.2 · p. 13

  • Takeaways & Limitations

    Temporal-mask pixel regeneration is presented as a route to creative video editing, independent of video length, camera distance, or subject identity.

    Source: §4.3 · p. 15

    Because synthesis uses only a monocular source image, VLOGGER has no access to occluded parts.

    Source: §4.3 · p. 15

Abstract

from arXiv · show

We propose VLOGGER, a method for audio-driven human video generation from a single input image of a person, which builds on the success of recent generative diffusion models. Our method consists of 1) a stochastic human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture that augments text-to-image models with both spatial and temporal controls. This supports the generation of high quality video of variable length, easily controllable through high-level representations of human faces and bodies. In contrast to previous work, our method does not require training for each person, does not rely on face detection and cropping, generates the complete image (not just the face or the lips), and considers a broad spectrum of scenarios (e.g. visible torso or diverse subject identities) that are critical to correctly synthesize humans who communicate. We also curate MENTOR, a new and diverse dataset with 3d pose and expression annotations, one order of magnitude larger than previous ones (800,000 identities) and with dynamic gestures, on which we train and ablate our main technical contributions. VLOGGER outperforms state-of-the-art methods in three public benchmarks, considering image quality, identity preservation and temporal consistency while also generating upper-body gestures. We analyze the performance of VLOGGER with respect to multiple diversity metrics, showing that our architectural choices and the use of MENTOR benefit training a fair and unbiased model at scale. Finally we show applications in video editing and personalization.

1 Introduction

VLOGGER targets person-agnostic generation of talking, moving humans from a single image, addressing prior methods’ limited control over bodies and identities. It combines audio-driven motion prediction with controlled video diffusion and evaluates the approach alongside a diverse training dataset and editing applications.

1 Introduction

  • Earlier approaches often focused on lip or face motion, with head cropping and limited generalization to new identities or visible bodies.The paper positions VLOGGER between dynamic video generation without identity or pose control and controllable image generation.Source: §1 · p. 3, Table 1 · p. 4
  • VLOGGER predicts stochastic audio-conditioned body motion and facial expressions, then uses spatial and temporal controls to generate video.It also uses temporal outpainting for videos of arbitrary length and supports editing regions such as lips or faces.Source: §1 · pp. 3–4
  • The paper presents VLOGGER for video editing and personalization, including inpainting selected regions such as lips or faces.It also analyzes the method’s stochasticity.Source: §1 · p. 4
  • VLOGGER outperforms previous state-of-the-art on three public benchmarks and shows low bias across evaluated human attributes.The method also generates head and upper-body motion, including diverse expressions and gestures.Source: §1 · pp. 3–4
  • MENTOR is a curated dataset one order of magnitude larger than existing datasets, with greater diversity and dynamic hand gestures.Its diversity includes skin tone, pose, viewpoint, speech, and body visibility.Source: §1 · pp. 3–4

2 Related Work

Related work spans audio-driven 3D motion, face-focused image generation, face reenactment, and general video generation. VLOGGER combines identity-conditioned human motion controls with temporal image diffusion to generate talking humans with body motion.

Audio-Driven Talking Face Generation

  • Audio-to-motion methods encode audio into 3D face or body motion, while VLOGGER targets photorealistic humans coherent with a target image [65].Earlier 3D-model approaches generate temporally coherent pose and shape parameters.Source: §2 · p. 4
  • Image-domain talking-face work expanded from mouth motion to head motion, gaze, and blinking, but body gestures remained constrained by data and coherent-video generation challenges.VLOGGER addresses this with a large-scale dataset and a pipeline for coherent face and upper-body motion.Source: §2 · pp. 4–5

Face Reenactment

  • Face reenactment transfers source-video motion to a target, but many methods require target-person frames for retraining and animation [96].VLOGGER instead considers diverse inputs that may include body and hair motion.Source: §2 · p. 5

Video Generation

  • Most video-diffusion extensions have limits in duration or resolution and do not explicitly focus on humans; VLOGGER adds spatiotemporal controls and iterative outpainting [2] [6] [24] [26] [35] [36] [72] [83].Its frame controls combine dense renders of a posed 3D body model with warped reference images.Source: §2 · p. 5

3 Method

VLOGGER uses a stochastic two-stage pipeline: audio becomes body-motion controls, then temporal image diffusion uses those controls and a reference image to generate identity-specific frames.

3 Method

  • VLOGGER maps audio to gaze, facial-expression, and 3D-pose controls, then uses a reference image and temporal image-to-image diffusion to generate corresponding identity-specific frames.Source: §3 · p. 6

3.1 Audio-Driven Motion Generation

  • The motion transformer predicts audio-driven face and body parameters and rasterizes posed bodies into dense controls; weighted temporal losses smooth head and hand motion while allowing more dynamic facial expressions.Source: §3.1 · pp. 6–8

3.2 Generating Photorealistic Talking and Moving Humans

  • A frozen image-diffusion encoder gains zero-initialized control layers and interleaved temporal convolutions, trained first on single frames and then on video clips.Source: §3.2 · p. 8
  • Cascaded diffusion raises 128 × 128 base videos to 256 × 256 or 512 × 512, while temporal outpainting extends generation to thousands of frames.Outpainting generates overlapping clips, trading overlap against quality and running time.Source: §3.2 · pp. 8–9

3.3 MENTOR Dataset

  • MENTOR contains 2.2K hours and 800K identities, with diverse skin tones, ages, viewpoints, and body visibility, after filtering videos with unstable backgrounds, incomplete detections, jitter, or poor audio.Source: §3.3 · p. 9

4 Experiments

Experiments evaluate VLOGGER on HDTF, TalkingHead-1KH, and MENTOR using image-quality, lip-sync, temporal-consistency, identity-preservation, and diversity measures. They report benchmark comparisons, ablate design choices, and show qualitative generation, editing, and personalization results.

4 Experiments

  • Evaluation uses HDTF, TalkingHead-1KH, and MENTOR; metrics cover image quality, lip sync, temporal smoothness, identity, and expression or gaze diversity.Body and hand-motion diversity is assessed qualitatively because the baselines do not consider gestures.Source: §4 · pp. 9–10

4.1 Ablation Study

  • Ablations favor residual motion prediction, temporal loss and classifier-free guidance, body controls, and temporal outpainting with 50% overlap.The training schedule of single-image training followed by video-layer fine-tuning also gives an additional performance boost.Source: §4.1 · pp. 10–11, Table 2 · p. 11
  • Dense body representations combined with warped reference views ease video generation and yield the best results among the tested 2D controls.The controls are compared using image-reconstruction metrics on MENTOR.Source: §4.1 · p. 10, Table 3 · p. 11

4.2 Quantitative Results

  • VLOGGER achieves the highest image quality and identity preservation, with expression diversity and temporal consistency close to ground truth and comparable lip-sync scores.These results are consistent across HDTF and TalkingHead-1KH.Source: §4.2 · pp. 11–12, Table 4 · p. 12
  • VLOGGER shows fairly low bias across evaluated perceived attributes, while other methods degrade across classes and do not generalize to visible torsos or hands.The comparisons include attributes such as skin tone, gender, and age on the MENTOR test set.Source: §4.2 · pp. 12–13, Figure 3 · p. 13

4.3 Qualitative Results

  • On in-the-wild images, VLOGGER generates varied facial expressions and inpaints occluded regions, with visible upper-body changes unlike cropped-head baselines.The qualitative comparison shows animated faces; gesture examples are presented separately.Source: §4.3 · p. 13, Figure 4 · p. 14
  • For the same subject and input speech, VLOGGER produces diverse videos, with hand-pose and facial-expression variation visible after one second.The figure summarizes pixel-color deviation across 24 generated videos.Source: Figure 5 · p. 14
  • Temporal inpainting supports expression edits such as changing the mouth or eyes, while video fine-tuning improves identity capture across expressions.Fine-tuning can help when the reference image shows closed eyes; the source synthesis itself uses a single image.Source: Figure 6 · p. 15, §4.3 · pp. 15–16, Figure 7 · p. 15

5 Conclusion

VLOGGER synthesizes variable-length human videos from a single image, conditioned on audio or text, using diffusion models with 3D head and body pose representations. It outperforms prior state of the art on talking-face generation and is more robust across diversity axes.

5 Conclusion

  • VLOGGER outperforms previous state-of-the-art on talking-face generation and is more robust across different diversity axes.Source: §5 · p. 16
  • VLOGGER extends control-based diffusion models with 3D human head and body pose representations to generate high-quality animations of variable length from a single image, conditioned on audio or text.Source: §5 · p. 16
  • The paper introduces a diverse, large-scale dataset one order of magnitude larger than previous ones and validates VLOGGER on it and multiple other repositories.Source: §5 · p. 16

Related papers

Loading 2403.08764v1…