PaperlayerView exact revision on arXiv

Source-linked AI summary

Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation

Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, Jingdong Wang

arXiv:2410.07718v2cs.CV

TL;DR

Long-duration, high-resolution portrait animation with consistent visual quality and temporal coherence remains challenging. Hallo2 extends Hallo using motion-frame augmentation, temporally aligned vector-quantized codes, and adjustable textual prompts alongside audio; the paper reports 4K portrait animation for durations reaching minutes or hours.

  • Problem

    Generating long-duration, high-resolution portrait videos with consistent visual quality and temporal coherence remains a challenge.

    Source: §2 · p. 4

  • Method

    Hallo2 combines patch-drop and Gaussian-noise augmentation of motion frames, temporally aligned vector-quantized latent codes, and adjustable textual prompts alongside audio.

    Source: §1 · p. 3, §4.1 · pp. 6–7

  • Results

    The authors report 4K portrait animation for durations extending up to ten minutes or several hours, with experiments on HDTF, CelebV, and Wild.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The approach combines audio and adjustable prompts for expression control with temporally coherent 4K output over extended durations.

    Source: §6 · p. 15

    A single reference image constrains expression and pose diversity, patch-drop may introduce artifacts, and 4K generation has substantial computational demands.

    Source: §5.4 · p. 15

Abstract

from arXiv · show

Recent advances in latent diffusion-based generative models for portrait image animation, such as Hallo, have achieved impressive results in short-duration video synthesis. In this paper, we present updates to Hallo, introducing several design enhancements to extend its capabilities. First, we extend the method to produce long-duration videos. To address substantial challenges such as appearance drift and temporal artifacts, we investigate augmentation strategies within the image space of conditional motion frames. Specifically, we introduce a patch-drop technique augmented with Gaussian noise to enhance visual consistency and temporal coherence over long duration. Second, we achieve 4K resolution portrait video generation. To accomplish this, we implement vector quantization of latent codes and apply temporal alignment techniques to maintain coherence across the temporal dimension. By integrating a high-quality decoder, we realize visual synthesis at 4K resolution. Third, we incorporate adjustable semantic textual labels for portrait expressions as conditional inputs. This extends beyond traditional audio cues to improve controllability and increase the diversity of the generated content. To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image animations enhanced with textual prompts. We have conducted extensive experiments to evaluate our method on publicly available datasets, including HDTF, CelebV, and our introduced "Wild" dataset. The experimental results demonstrate that our approach achieves state-of-the-art performance in long-duration portrait video animation, successfully generating rich and controllable content at 4K resolution for duration extending up to tens of minutes. Project page https://fudan-generative-vision.github.io/hallo2

1 INTRODUCTION

Hallo2 extends Hallo toward long-duration, 4K audio-driven portrait animation with adjustable textual expression prompts. It uses augmented preceding frames for motion, temporally aligned vector-quantized codes for high-resolution synthesis, and reports generation at 4K for durations reaching minutes or hours.

1 INTRODUCTION

  • Parallel long-video generation can blur or distort expressions under continuity constraints, while incremental generation risks accumulating errors from preceding frames.Source: §1 · p. 2, Figure 2 · p. 3
  • Patch-drop corruption and Gaussian noise on conditional frames help preserve motion while keeping appearance anchored to the reference portrait during long videos.The method follows incremental generation, using preceding frames for motion dynamics rather than appearance.Source: §1 · p. 2
  • Temporal alignment of vector-quantized latent-code prediction enables smooth code transitions that support temporally coherent 4K synthesis with a high-quality decoder.Source: §1 · p. 3
  • Adjustable textual prompts supplied at different intervals condition generation alongside audio and can adjust facial expressions and head poses.Source: §1 · p. 3
  • Hallo2 is reported as the first portrait-animation method to achieve 4K resolution for durations extending up to ten minutes or several hours, with adjustable prompts for facial control.Experiments cover HDTF, CelebV, and the introduced Wild dataset.Source: §1 · p. 3

2 RELATED WORK

Related work spans portrait animation, long-video generation, and discrete latent representations. Hallo2 builds on Hallo while addressing long-duration portrait consistency and using temporally aligned vector quantization for 4K synthesis.

Video Diffusion Models

  • Video-generation work includes latent diffusion systems and methods targeting text-video alignment, motion guidance, personalized animation, and generation efficiency.Source: §2 · p. 3

Portrait Image Animation

  • Portrait-animation methods have advanced realism, synchronization, expressiveness, and identity preservation, but long-duration, high-resolution videos with consistent quality and temporal coherence remain challenging.Hallo2 builds on Hallo to address this gap.Source: §2 · p. 4

Long-Term and High-Resolution Video Generation

  • Long-video methods use approaches including generative transitions, semantic motion prediction, autoregressive generation, distributed inference, and global-local feature integration; Hallo2 instead applies patch-drop and Gaussian-noise augmentation to portrait animation.Source: §2 · p. 4

Discrete Prior Representations

  • Discrete latent dictionaries support image and video generation, restoration, and conditional synthesis; Hallo2 uses vector-quantized codes with temporal alignment for 4K coherence.Source: §2 · p. 4

3 PRELIMINARIES

Latent diffusion models perform diffusion and denoising in a compressed latent space, reducing computational complexity while maintaining image quality. Cross-attention integrates motion conditions to guide denoising.

3.1 LATENT DIFFUSION MODELS

  • Diffusion and denoising in a compressed latent space reduce computational complexity while maintaining image quality.A pretrained VAE encodes input images into lower-dimensional latent representations.Source: §3.1 · pp. 4–5
  • Forward diffusion adds Gaussian noise to the encoded latent over T steps, while the accumulated noise drives its distribution toward a standard normal.The schedule uses α_t = 1−β_t and the cumulative product ᾱ_t.Source: §3.1 · p. 5
  • A noise-prediction network is trained to estimate the noise in each latent, then iteratively denoises a random Gaussian latent for decoding into an image.The training objective is expected mean squared error, with ω(t) balancing contributions across timesteps.Source: §3.1 · p. 5

3.2 INCORPORATING MOTION CONDITIONS VIA CROSS-ATTENTION

  • Cross-attention processes noisy latents alongside motion conditions, including reference-image, audio, and textual embeddings, to guide generation.The attention weights focus on relevant condition components, supporting control over expressions, lip movements, and head poses.Source: §3.2 · p. 5

4 METHOD

Hallo2 extends portrait animation with long-duration motion conditioning, temporal codebook prediction for high-resolution video, and textual control of expressions and motion. It derives appearance from a reference image while using preceding frames for motion.

4 METHOD

  • Hallo2 combines reference-image appearance with preceding-frame motion cues, patch-drop and Gaussian-noise augmentation, temporal codebook prediction, and textual conditioning.The method targets long-duration, high-resolution portrait animation with audio- and text-based control.Source: §4 · p. 6

4.1 LONG-DURATION ANIMATION

  • Patch dropping and Gaussian noise corrupt preceding-frame appearance information while retaining motion cues, helping stabilize appearance and temporal dynamics over long sequences.The corrupted motion frames are used as diffusion-model conditioning inputs.Source: §4.1 · pp. 6–7

4.2 HIGH-RESOLUTION ENHANCEMENT

  • Temporal alignment in codebook prediction captures frame-to-frame dependencies for high-resolution video; a lightweight alignment module is favored when super-resolution training data are sparse.The alternative directly uses diffusion latents but requires end-to-end training of the full enhancement module.Source: §4.2 · p. 7

4.3 TEXTUAL PROMPT CONTROL

  • CLIP text embeddings generate adaptive scaling and shifting parameters that condition denoising features, enabling fine-grained control of synthesized expressions and motion.Adaptive layer normalization is placed between cross-attention and audio attention.Source: §4.3 · p. 8

4.4 NETWORK

  • The denoising U-Net combines reference-image, corrupted-motion, audio, and text conditioning; two-stage training and a high-resolution enhancement module complete the generation pipeline.Inference accepts a reference image, driving audio, optional text, and augmented motion frames before enhancing the generated frames.Source: §4.4 · pp. 8–9

5 EXPERIMENTS

The experiments compare Hallo2 with existing methods on three datasets and evaluate its augmentation, high-resolution enhancement, and textual controls. They also identify constraints involving expression and pose diversity, artifacts, and the compute required for 4K generation.

5.1 EXPERIMENTAL SETUPS

  • Experiments use HDTF, CelebV, and Wild, and compare methods on four-minute videos at 512 × 512 pixels without high-resolution enhancement.The Wild dataset contains approximately 155.9 hours of video across 2019 clips; evaluation includes FID, FVD, Sync-C, Sync-D, and E-FID.Source: §5.1 · pp. 9–11

5.2 COMPARISON WITH STATE-OF-THE-ART

  • Across HDTF, CelebV, and Wild, the method reports FID/E-FID of 16.616/6.702, 37.944/14.682, and 24.072/33.316, respectively.The comparisons also report stable performance as inference duration increases on HDTF and deterioration in existing methods on CelebV.Source: §5.2 · pp. 11–12
  • The method supports varied portrait styles, including oil paintings, anime images, and portraits from generative models.Figure 9 presents this qualitative comparison.Source: §5.2 · p. 12, Figure 9 · p. 12

5.3 ABLATION STUDIES

  • Combined patch-drop and Gaussian-noise augmentation achieves FID 37.944, FVD 477.412, and Sync-C 6.928, outperforming either strategy alone on these reported measures.Patch size 1 is used in the implementation; varying the drop rate shows that 0.25 achieves FID 38.518 and FVD 491.338.Source: §5.3 · p. 13
  • Super-resolution improves animation quality, while combining it with temporal alignment enhances fidelity, reduces artifacts, and sharpens images.The comparison is qualitative.Source: §5.3 · p. 13
  • Textual prompts allow more precise control of facial expressions and emotional nuances, while attention maps associate appearance with the reference image and motion with facial regions.The prompt comparison reports improved responsiveness to specified emotional states.Source: §5.3 · pp. 13–14

5.4 LIMITATIONS AND FUTURE WORK

  • A single reference image constrains expression and pose diversity, patch drop may introduce artifacts, and 4K generation requires substantial computation.The authors identify optimization and hardware acceleration as needs for real-time applications.Source: §5.4 · p. 15

6 CONCLUSION

The conclusion presents Hallo enhancements for longer-duration, 4K portrait animation with adjustable textual prompts. Experiments on publicly available datasets validate the method's effectiveness.

6 CONCLUSION

  • The method extends animation duration to tens of minutes while maintaining 4K output.The conclusion describes this as addressing limitations of existing methods.Source: §6 · p. 15
  • Patch-drop and Gaussian-noise augmentation are presented as supporting identity consistency and reducing appearance contamination.Source: §6 · p. 15
  • Vector quantization, temporal alignment, and adjustable semantic textual prompts support temporally consistent 4K video and control of expressions and motion dynamics.The conclusion reports validation through experiments on publicly available datasets.Source: §6 · p. 15

Related papers

Loading 2410.07718v2…