PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation

Haojie Zhang, Zhihao Liang, Ruibo Fu, Bingyan Liu, Zhengqi Wen, Xuefei Liu, Jianhua Tao, Yaling Liang

arXiv:2411.16748v5cs.CV

TL;DR

Long-duration talking-video synthesis struggles to preserve quality, identity, temporal coherence, and efficiency as sequences grow. LetsTalk combines a noise-regularized memory bank, compressed spatiotemporal diffusion Transformer, and multimodal fusion, achieving strong benchmark performance and efficient generation. Its scope remains bounded by challenges under difficult reference conditions and long temporal horizons.

  • Problem

    Long-duration talking-video synthesis lacks reliable preservation of visual quality, portrait consistency, temporal coherence, and computational efficiency as video length increases.

    Source: §I · p. 1

  • Method

    LetsTalk uses a noise-regularized memory bank, 32× deep compression autoencoder, linear-attention spatiotemporal Transformer, and multimodal fusion for long-duration generation.

    Source: §I · pp. 1–2, A. Efficient Linear Transformer with… · p. 4, B. Long-duration Generation with… · p. 4

  • Results

    LetsTalk-L achieves the lowest HDTF FID (31.761) and FVD (219.119), outperforming Hallo2 by 28.3% and 9.3%, while also achieving the lowest CelebV-HQ FID (7.610) and FVD (33.175).

    Source: C. Quantitative Results · p. 7

  • Takeaways & Limitations

    LetsTalk provides realistic, temporally coherent, diverse long-duration talking videos with a favorable quality-efficiency trade-off.

    Source: §I · p. 2, §V · p. 10

    Challenges remain under difficult reference conditions and over long temporal horizons.

    Source: D. Discussion and Future Work · p. 14

Abstract

from arXiv · show

Long-duration talking video synthesis faces enduring challenges in achieving high video quality, portrait consistency, temporal coherence, and computational efficiency. As video length increases, issues such as visual degradation, portrait drift, temporal artifacts, and error accumulation become increasingly problematic, severely affecting the realism and reliability of the results. To address these challenges, we present LetsTalk, a diffusion transformer framework equipped with multimodal guidance and a novel memory bank mechanism, explicitly maintaining contextual continuity and enabling robust, high-quality, and efficient generation of long-duration talking videos. In particular, LetsTalk introduces a noise-regularized memory bank to alleviate error accumulation and sampling artifacts during extended video generation. To further improve efficiency and spatiotemporal modeling, LetsTalk employs a deep compression autoencoder and a spatiotemporal-aware transformer with linear attention for effective multimodal fusion. We systematically analyze three fusion schemes and show that combining deep (Symbiotic Fusion) for portrait features and shallow (Direct Fusion) for audio achieves superior visual realism and precise speech-driven motion, while preserving diversity of movements. Extensive experiments demonstrate that LetsTalk establishes new state-of-the-art in generation quality, producing temporally coherent and realistic talking videos with enhanced diversity and liveliness, and maintains remarkable efficiency with 8x fewer parameters than previous approaches.

I. INTRODUCTION

Long-duration talking-head generation must balance visual quality, portrait consistency, temporal coherence, expressiveness, and computational efficiency. LetsTalk addresses these challenges with a noise-regularized memory bank, efficient spatiotemporal modeling, and multimodal fusion, achieving realistic and coherent videos with favorable quality-efficiency trade-offs.

  • Longer videos amplify visual degradation, portrait drift, accumulated errors, and temporal artifacts, undermining realism and reliability.Source: §I · p. 1
  • LetsTalk introduces a noise-regularized memory bank that preserves prior context while simulating inference-time noise accumulation during training.The mechanism is designed to mitigate error accumulation and sampling artifacts across extended video sequences.Source: §I · pp. 1–2
  • LetsTalk combines a 32× deep compression autoencoder with temporal and spatial modules using Linear Attention for efficient long-video generation.The architecture reduces latent token counts while modeling intra-frame and inter-frame coherence without excessive computational cost.Source: §I · pp. 1–2
  • Symbiotic Fusion for portrait features and Direct Fusion for audio provide superior visual quality, temporal coherence, and speech-driven guidance.The paper compares Direct, Siamese, and Symbiotic Fusion according to integration depth, conditional state, and parameter sharing.Source: §I · p. 2
  • Experiments report realistic, coherent, diverse long-duration talking videos with favorable quality-efficiency trade-offs.LetsTalk-B matches Hallo’s performance with 8× fewer parameters, as shown in the HDTF comparison.Source: §I · p. 2, Figure 1 · p. 2

II. RELATED WORK

Diffusion Transformers have become a scalable basis for controllable video generation, while talking-head methods increasingly combine audio, portrait, and other modalities. LetsTalk extends this line with efficient multimodal control and a memory mechanism aimed at sustained long-duration coherence.

A. Video Generation with Diffusion Transformers

  • Diffusion video generation evolved from space-time UNets toward scalable DiT architectures with temporal attention and controllable conditioning.LetsTalk adopts a DiT-based framework for multimodal audio-and-portrait control.Source: A. Video Generation with Diffusion… · p. 2

B. Audio-driven Portrait Image Animation

  • Portrait animation progressed from lip synchronization and 3D information toward diffusion-based systems supporting higher fidelity, expression, and controllability.Recent methods also explore natural-language guidance, expressive faces, and emotion experts.Source: B. Audio-driven Portrait Image Animation · pp. 2–3
  • LetsTalk targets efficiency beyond adopting a Transformer backbone by combining linear attention, deep compression, and multimodal guidance.The framework integrates portrait and audio conditions while addressing computational cost in long-duration synthesis.Source: B. Audio-driven Portrait Image Animation · p. 3, Figure 2 · p. 3
  • Long-video methods use 3DMMs, sliding windows, noise rescheduling, short-term memory, or temporal training strategies, but sustained coherence remains difficult.LetsTalk draws on these directions while introducing a trainable noise-regularized memory mechanism for long-range context.Source: C. Long-Duration Video Generation · p. 3

III. METHODOLOGY

LetsTalk combines efficient spatiotemporal processing with a noise-regularized memory bank to support coherent, scalable long-duration video generation. Its training strategy simulates inference-time noise and uses controlled corruption to improve robustness.

A. Efficient Linear Transformer with Deep Autoencoder

  • The framework combines a deep compression autoencoder, interleaved spatial-temporal attention, and linear attention for efficient long-duration video processing.The autoencoder downsamples images by 32×, while the interleaved design preserves global spatial and temporal context.Source: A. Efficient Linear Transformer with… · pp. 3–4

B. Long-duration Generation with Noise-Regularized Memory

  • The training noise range samples diffusion timesteps t ∈[0, 50], covering observed error magnitudes near t ≈17 while retaining a safety margin.The range is intended to cover realistic drift without adding excessive noise that could degrade feature semantics.Source: Figure 4 · p. 4, B. Long-duration Generation with… · pp. 4–5
  • The memory bank stores patches from previously generated clips and injects them into current noisy latents through Symbiotic Fusion to preserve long-range continuity.This design targets smooth transitions and consistency across video segments.Source: B. Long-duration Generation with… · p. 4
  • Noise-regularized training perturbs the memory bank with controlled Gaussian noise to bridge the gap between ground-truth training memories and generated inference memories.The perturbation simulates accumulated inference-time residual noise and sampling artifacts.Source: B. Long-duration Generation with… · p. 4
  • Randomly masking 25% of memory-bank spatial patches further trains the model to correct artifacts and maintain temporal dependencies.This augmentation is presented as an additional robustness mechanism for stable long-duration synthesis.Source: B. Long-duration Generation with… · p. 5

C. Systematic Analysis and Selection of Fusion Schemes

The paper categorizes Direct, Siamese, and Symbiotic Fusion by their conditioning mechanisms and selects them according to modality-specific needs. The resulting design favors shared self-attention for portrait structure and cross-attention for audio guidance.

Fusion scheme taxonomy

  • Direct Fusion injects conditions through cross-attention, offering computational efficiency but limiting interaction to the cross-attention bottleneck.It treats the condition as context while keeping the generation backbone focused on the target.Source: C. Systematic Analysis and Selection… · p. 5
  • Siamese Fusion uses a parallel condition encoder for precise control, but its dedicated parameters increase model size and training overhead.The scheme is associated with methods including AnimateAnyone, Hallo, and EMO.Source: C. Systematic Analysis and Selection… · p. 5
  • Symbiotic Fusion concatenates condition tokens with latent inputs so shared self-attention learns a unified representation with strong alignment and parameter efficiency.The trade-off is greater optimization difficulty than cross-attention-based conditioning.Source: C. Systematic Analysis and Selection… · p. 5

Design selection

  • The selected asymmetric design uses Symbiotic Fusion for portrait consistency and Direct Fusion for semantic audio guidance and motion diversity.Cross-attention is treated as a lighter path for heterogeneous audio while preserving the visual backbone’s spatial integrity.Source: C. Systematic Analysis and Selection… · p. 6

D. Guidance on Reference Portrait and Audio

LetsTalk applies deep Symbiotic Fusion to reference portraits and Direct Fusion to audio, aligning each modality with a different conditioning mechanism. Experiments use identity-disjoint HDTF and CelebV-HQ data to assess portrait consistency and generalization.

Reference portrait guidance

  • Reference portraits are encoded into static feature maps, replicated across frames, and concatenated with noisy video latents for shared self-attention at every layer.This joint modeling keeps character appearance robust throughout denoising.Source: D. Guidance on Reference Portrait and… · p. 6

Audio guidance

Experimental setup

  • The evaluation uses HDTF and CelebV-HQ, with identity-disjoint splits that reserve 200 clips for testing unseen-identity generalization.The datasets contain diverse identities and recording environments after filtering low-quality or unsuitable videos.Source: A. Experimental Setups · p. 6, Table II · p. 6

Guidance outcomes

  • Qualitative comparisons report closer audio-animation alignment, stronger expressiveness and diversity, and strict portrait consistency on the evaluated datasets.The portrait and audio behaviors are attributed respectively to Symbiotic Fusion and Direct Fusion.Source: Figure 7 · p. 7

B. Additional Implementation Details

The experiments compare LetsTalk with substantially larger portrait-animation systems under standardized evaluation settings. LetsTalk achieves the strongest reported FID and FVD results on both datasets and maintains quality and identity stability in long-duration generation.

Implementation details

  • The implementation uses an end-to-end training strategy with Base and Large variants, while inference generates 16-frame 512×512 clips using 250-step DDPM sampling.The comparison includes models ranging from 138M to 8B parameters.Source: B. Additional Implementation Details · p. 7

Quantitative results

  • LetsTalk-L achieves the lowest HDTF FID of 31.761 and FVD of 219.119, outperforming Hallo2 by 28.3% and 9.3%, respectively.All methods are evaluated at the same resolution using the first validation clip.Source: C. Quantitative Results · p. 7
  • On CelebV-HQ, LetsTalk achieves the lowest reported FID of 7.610 and FVD of 33.175.The paper interprets these results as stronger realism and temporal consistency.Source: C. Quantitative Results · p. 7, Table IV · p. 7

Long-duration results

  • A continuous approximately four-minute comparison shows LetsTalk maintaining visual quality and identity stability while Hallo2 develops later-stage error accumulation artifacts.The sequence contains approximately 6,800 frames.Source: Figure 8 · p. 8

Qualitative results

  • Qualitative results show close alignment to ground-truth facial and lip motion together with expressive and diverse generated movements.These observations are reported across HDTF and CelebV-HQ visualizations.Source: D. Qualitative Results · pp. 7–8

E. Long-Duration Generation Assessment

LetsTalk is evaluated for long-duration stability, efficiency, and multimodal fusion. Results support noise-regularized training, Patch Drop, and asymmetric fusion as mechanisms for balancing fidelity, temporal consistency, portrait quality, and motion diversity.

Ablation Study

  • Noise-Regularized Training achieves the best FID of 32.746, while adding Patch Drop produces the lowest FVD of 251.684 on 100 consecutive clips.The final model combines both strategies because Patch Drop trades a slightly higher FID of 34.821 for stronger temporal consistency.Source: E. Long-Duration Generation Assessment · p. 8, Table VI · p. 9

Comparison with State-of-the-Art

  • Beyond 4 minutes, LetsTalk preserves visual fidelity and portrait consistency without the temporal drift and artifact accumulation observed in Hallo2.Hallo2 develops forehead texture interference and identity distortion toward frame 6800, whereas LetsTalk remains consistent throughout the sequence.Source: E. Long-Duration Generation Assessment · p. 8

Long-Duration Consistency

  • LetsTalk-L maintains stronger long-duration robustness than Hallo2, with a slower FID and FVD decline as consecutively generated clips increase.The comparison evaluates performance over increasing generation horizons and attributes the advantage to reduced long-duration error accumulation.Source: Figure 9 · p. 9

Efficiency Comparison

  • LetsTalk-B achieves comparable or better visual quality with fewer parameters and faster inference, while LetsTalk-L further improves FID with reasonable efficiency.These results indicate a quality–efficiency trade-off favorable to the proposed model variants.Source: F. Efficiency Comparison · pp. 8–9

Multimodal Fusion

  • Symbiotic Fusion best supports portrait consistency, whereas Direct Fusion better preserves audio-animation alignment and motion diversity.The asymmetric design uses tighter visual interaction for portrait features and explicit conditioning for audio.Source: G. Ablation Study · p. 9, Table VII · p. 9, Table VIII · p. 9
  • Without audio guidance, high-compactness Siamese and Symbiotic Fusion restrict movement diversity, while Direct Fusion preserves it.This result links fusion compactness with the diversity of generated portrait animation.Source: Figure 10 · p. 9, G. Ablation Study · p. 9

H. User Study

A 20-participant user study compares LetsTalk with four talking-video methods across five perceptual criteria. LetsTalk-L leads all categories, especially portrait and long-duration consistency.

Study Design

  • The evaluation uses 30 diverse audio samples, including sequences longer than 10 seconds, with 20 participants ranking five perceptual criteria.Participants included researchers, graduate students, and general users, and could replay samples before ranking.Source: H. User Study · pp. 9–10

User Preference Evaluation

  • LetsTalk-L leads all five perceptual categories and substantially outperforms the runner-up Hallo2 in portrait and long-duration consistency.The study compares visual realism, lip synchronization, portrait consistency, motion naturalness, and long-duration consistency using randomized, hidden-name samples.Source: H. User Study · pp. 9–10, Figure 11 · p. 10
  • LetsTalk-L receives 71.2% preference for long-duration consistency and 67.7% for portrait consistency.These are the strongest reported preference results and are identified as dominating the runner-up Hallo2.Source: H. User Study · p. 10
  • EchoMimic is competitive in lip synchronization at 27.8%, while Hallo2 follows in motion naturalness at 27.2%.These are secondary category-specific results despite LetsTalk-L’s overall lead.Source: H. User Study · p. 10

APPENDIX

The appendix describes the diffusion formulation and its Diffusion Transformer extension. The framework trains a noise-prediction network while adding adaptive temporal integration into spatial representations.

Diffusion Formulation

  • The diffusion process corrupts data with Gaussian noise, and the model learns the reverse transition pθ(x_t−1|x_t).The forward process is parameterized by predetermined noise-schedule hyperparameters, while the reverse process uses neural-network estimates of the transition distribution.Source: A. Preliminaries · p. 12
  • Reparameterizing the reverse mean as a noise-prediction network enables training with mean-squared error between predicted and sampled noise.The formulation also optimizes the learnable reverse covariance with the full variational objective.Source: A. Preliminaries · p. 12

Diffusion Transformer

  • The Diffusion Transformer extends patch-based Vision Transformer processing to support multimodal fusion in talking-head generation.Its sequential patch representation is described as supporting multimodality efficiently.Source: A. Preliminaries · p. 12

Adaptive Integration

  • Adaptive temporal integration adds a transformed temporal-attention signal to spatial features through a learnable scalar and fully connected layer.This design enables temporal dynamics to be fused into spatial representations between attention layers.Source: 1) Adaptive Integration · p. 12

2) Audio Encoder:

LetsTalk extracts frame-aligned audio features with wav2vec and projects them into fixed context tokens for transformer conditioning. The appendix also describes portrait guidance, qualitative generalization, and aligned visual comparisons.

Audio Encoder

  • Audio is resampled to 16 kHz, extracted from wav2vec’s last 12 layers, aligned to video frames, and represented using sliding audio windows.These steps are intended to preserve richer semantic information and audio continuity.Source: 2) Audio Encoder · p. 12
  • The audio projection module flattens extracted features, selects 32 context tokens, and maps them into the latent-noise feature space.This produces audio representations suitable for transformer tokenization.Source: 2) Audio Encoder · p. 12

Siamese Transformer

  • The Siamese Fusion scheme uses a parallel spatial transformer for the static portrait and a lightweight transformer for audio to preserve identity guidance.The portrait branch removes temporal attention because the reference image lacks temporal dynamics.Source: 3) Siamese Transformer · p. 12

Generalization

Qualitative Evaluation

3) Impact of Reference Head Pose:

LetsTalk is most reliable with frontal or near-frontal reference portraits, while large pose deviations reduce visible facial geometry and weaken fine-grained identity preservation.

Impact of Reference Head Pose

  • Frontal or near-frontal reference portraits provide the most reliable operating condition for LetsTalk.Large yaw or pitch angles reduce overlap between the static portrait prior and generated target frames.Source: 3) Impact of Reference Head Pose · p. 13
  • Large yaw or pitch can preserve coarse identity while losing local details around contours, cheeks, or the far-side eye.Incomplete facial geometry makes portrait tokens less reliable for identity supervision.Source: 3) Impact of Reference Head Pose · p. 14

Robustness to Harsh Lighting

  • Harsh lighting may cause unstable skin-texture recovery or local brightness inconsistency, despite broad stability under moderate illumination changes.Strong backlighting, high-contrast shadows, and saturated highlights can be encoded with portrait features and propagated during denoising.Source: 4) Robustness to Harsh Lighting · p. 14

Influence of Reference Expression

  • Extreme reference expressions can bias generated frames toward expression-specific texture cues rather than fully neutral portrait consistency.Exaggerated smiles or tightly closed eyes are especially challenging.Source: 5) Influence of the Reference Expression · p. 14

Impact of Partial Occlusions

  • Partial occlusions can introduce local artifact accumulation or identity ambiguity in corresponding facial regions.The portrait branch and memory-guided generation depend on visible appearance cues.Source: 6) Impact of Partial Occlusions · p. 14

D. Discussion and Future Work

LetsTalk remains strong overall but is not fully stable under difficult reference conditions or extended temporal horizons. The discussion attributes these weaknesses to perturbation-specific appearance and identity disruptions, plus residual long-range error accumulation.

Discussion and Future Work

  • LetsTalk achieves strong overall performance, but challenges remain under difficult reference conditions and over long temporal horizons.The discussion frames these limitations as directions for future improvement.Source: D. Discussion and Future Work · p. 14

Robustness Comparisons

  • Large pose changes weaken side-region identity guidance, harsh lighting destabilizes local appearance, and extreme expressions bias expression-specific textures.These perturbations affect the model through incomplete geometry, illumination entanglement, and reference-expression cues, respectively.Source: 1) Challenging Reference Conditions · p. 14

Long-Duration Consistency

  • Fast motion, abrupt phoneme transitions, lighting variation, and occlusion changes can still cause flickering, texture softness, or short-term drift over long horizons.Small local errors may accumulate despite memory retrieval and noise regularization.Source: 2) Long-Duration Consistency · p. 14

Future Work

  • Future improvements include more robust portrait representations, selective memory retrieval and updates, and stronger disentanglement of static and dynamic factors.These directions target sensitivity to pose, illumination, expression, occlusion, and unreliable historical cues.Source: 3) Future Work · p. 16

Related papers

Loading 2411.16748v5…