PaperlayerView exact revision on arXiv

Source-linked AI summary

Avatar V: Scaling Video-Reference Avatar Video Generation

Benjamin Liang, Ce Chen, Desmond Lin, Ivan Somov, Jiajun Zhao, Jiewei Yuan, Jingfeng Zhang, Junhao Huang, Nik Nolte, Pedram Haqiqi, Penghan Wang, Rong Yan, Rui Zhang, Sam Prokopchuk, Sivan Wang, Viktor Goriachko, Yi Ren, Yuanming Li, Yutao Chen, Zhenhui Ye, Zhibin Hong, Zilong Nie, Zujin Guo

arXiv:2606.13872v1cs.CV

TL;DR

Faithfully generating production-quality talking avatars that preserve a person’s appearance, expressions, and talking style across diverse scenes remains challenging because static-image references miss dynamic identity information. Avatar V conditions directly on full reference-video tokens to model static and behavioral identity, achieving state-of-the-art cross-scene performance across evaluated dimensions and outperforming leading systems in automated metrics and human evaluation.

  • Problem

    Existing avatar systems largely rely on single static images, limiting representation of varied views, fine-grained facial details, and characteristic talking styles.

  • Method

    Avatar V conditions directly on a reference video’s full token sequence, using attention to capture static appearance and dynamic behavioral patterns without fixed-size identity bottlenecks.

  • Results

    Avatar V achieves state-of-the-art performance across all evaluated dimensions on a cross-scene benchmark, outperforming leading systems in automated metrics and human evaluation.

  • Takeaways & Limitations

    Video-reference conditioning enables Avatar V to preserve both a target person’s appearance and talking style in 1080p avatar videos of unlimited duration.

  • Takeaways & Limitations

    At scale, the data-processing pipeline outgrew Ray’s centralized coordination architecture, requiring a purpose-built replacement.

Abstract

from arXiv · show

Generating avatar videos that are not merely visually similar to a target individual but behaviorally recognizable, faithfully reproducing their talking rhythm, gestural tendencies, and expression dynamics, remains an open challenge. Existing methods predominantly condition on single static images, which provide insufficient identity information and cannot capture dynamic motion traits, while standard pixel-level objectives underserve the perceptually critical facial regions that determine avatar fidelity. We present Avatar V, a production-scale framework that addresses these limitations through video-reference-conditioned identity modeling. Rather than compressing identity into fixed-size embeddings, the model conditions directly on the full token sequence of a reference video, learning to reproduce both static identity attributes (facial geometry, skin texture) and dynamic behavioral patterns (talking rhythm, micro-expressions) through attention over the reference context. We introduce Sparse Reference Attention, an asymmetric mechanism achieving linear-complexity conditioning on arbitrarily long references; a motion representation stream enabling closed-loop talking style transfer; and an identity-aware super-resolution refiner inheriting the full reference conditioning. These are supported by a data engine curating 100M+ training clips from 50M raw videos, and a five-stage training pipeline with flow matching pre-training, personality fine-tuning, two-phase distillation (>10x acceleration), and RLHF alignment, deployed across thousands of GPUs. Avatar V generates 1080p videos of unlimited duration, achieving state-of-the-art identity preservation, lip synchronization, and generation quality on our cross-scene benchmark, consistently outperforming leading systems including Seedance 2.0, Kling O3 Pro, Veo 3.1, and OmniHuman 1.5 in both automated metrics and human evaluation.

1 Introduction

Avatar V addresses the open challenge of production-quality talking avatars by conditioning directly on full reference videos to reproduce both identity and behavioral style. Its architecture, data engine, training pipeline, and production deployment target high-fidelity generation across diverse scenes.

  • Motivation: Existing talking-avatar systems struggle with shallow identity representations, decoupled appearance and motion style, and insufficient supervision of critical facial regions.Static image conditioning causes identity drift and misses characteristic talking styles, while uniform pixel losses undertrain lips, teeth, expressions, and gaze.
  • Core approach: Avatar V conditions on the full token sequence of an arbitrary-length reference video, jointly modeling static identity attributes and dynamic behavioral patterns.Sparse Reference Attention lets generation tokens attend to all reference tokens while reference tokens self-attend, reducing conditioning complexity.
  • Core approach: A dedicated motion representation stream provides a closed-loop target and conditioning signal for learning each individual’s talking rhythm, expressions, and gestural dynamics.Joint optimization unifies motion modeling and generation for talking-style transfer.
  • Core approach: An identity-aware super-resolution refiner inherits full reference conditioning and uses identity and audio signals to recover fine-grained facial details with sparse temporal attention.The refiner is designed for practical high-resolution inference and addresses details lost at base resolution.
  • System and training: Over 50M raw videos support a three-tier data engine, while a five-stage training pipeline includes personality SFT, two-phase distillation for over 10× acceleration, and reinforcement learning from human feedback.The data engine links same-identity clips across distinct scenes, and auxiliary losses target identity, motion, lip-sync, and perceptual fidelity.
  • Results and deployment: Avatar V achieves state-of-the-art performance on a cross-scene benchmark, consistently outperforming Seedance 2.0, Kling O3 Pro, Veo 3.1, and OmniHuman 1.5 in automated metrics and human evaluation.The full production pipeline is deployed across 5,000+ GPUs with inference and multi-cloud infrastructure optimizations.

2 Model Design

Avatar V is an audio-driven DiT system that conditions directly on full user-video token sequences to preserve appearance, expressions, and talking style. Its design combines sparse reference attention, motion representations, human-aware losses, and identity-aware high-resolution refinement.

  • Core Architecture: Avatar V uses a flow-matching Diffusion Transformer that takes a short user video, target audio, and scene text to generate high-fidelity avatar videos.The system is designed to reproduce the target person’s appearance, expressions, and talking style.
  • Reference Conditioning: Full reference-video token sequences provide identity context at every transformer layer, preserving fine-grained appearance, expression, and motion patterns without fixed-size identity embeddings.The model treats personality embedding as a video-reference conditioning problem rather than compressing identity into low-dimensional representations.
  • Reference Conditioning: Sparse Reference Attention exploits dynamic features from long reference videos while avoiding the prohibitive quadratic cost of concatenating all reference and generation tokens.It is introduced as a structured sparsity mechanism for computationally tractable video-reference identity conditioning.
  • Motion Modeling: The motion representation stream models talking style as temporal variation in facial movements, mouth shapes, and head gestures during speech.It addresses speaker-specific articulation differences that can occur even for the same phoneme sequence.
  • Training Objectives: Human-aware auxiliary losses supplement pixel-level diffusion loss with semantic supervision for identity, talking style, and micro-expressions.These features occupy a small fraction of total pixel variation but are important for identity perception.
  • High-Resolution Refinement: The Super-Resolution Refiner inherits video-reference conditioning, audio features, and motion representations to enhance resolution while preserving facial identity consistency.It uses the low-resolution output as an additional conditioning signal alongside high-resolution noise.

3 Training Strategy

Avatar V uses a progressive five-stage training strategy that moves from general video generation through audio conditioning and identity-aware personality fine-tuning to distillation and human-feedback alignment. This curriculum targets temporal dynamics, identity preservation, efficient inference, and perceptual quality.

  • Training curriculum: The five-stage pipeline comprises text-to-video pre-training, audio-to-video pre-training, personality SFT, distillation, and RLHF.The stages progressively develop general video understanding, identity-specific personality embedding, efficient inference, and human-preference alignment.
  • Foundational pre-training: T2V pre-training establishes foundational video generation, progressively scaling resolution and duration while jointly training text-to-video and image-to-video tasks.Rectified flow matching provides the training objective, with a logit-normal timestep distribution and distributed Muon plus AdamW optimization.
  • Audio conditioning: Audio-to-video pre-training adapts the T2V checkpoint to first-frame and driving-audio conditioning, producing temporally coherent continuations with synchronized lip movements and natural head motion.This stage introduces audio cross-attention for audio-conditioned avatar generation.
  • Identity specialization: Personality SFT trains on same-identity-different-scene data with reference videos, activating Sparse Reference Attention and motion pathways for identity-aware avatar generation.The model learns to extract and use identity information from video references during fine-tuning.
  • Inference acceleration: Two-phase distillation removes multi-stream CFG overhead and reduces denoising steps, preserving generation quality while lowering inference cost.CFG distillation compresses multiple guidance streams into one forward pass; DMD then uses student, fake-teacher, and frozen real-teacher models.
  • Human-preference alignment: RLHF aligns the final model with human perceptual preferences using identity fidelity, motion naturalness, and visual quality rewards with GRPO.The policy optimization is adapted for flow matching, with KL regularization against the pre-RLHF model.

4 Inference … 4.3 VideoRef Context Caching and Sparse Attention

Avatar V’s inference pipeline combines a distilled 24-step model with staged preprocessing, chunk-based generation, identity-aware super-resolution, and streaming decode. Long-form synthesis is enabled by prefix-connected chunks, while caching and sparse attention reduce repeated computation over video-reference context.

  • 4 Inference: 24 denoising steps after two-phase distillation enable practical-latency inference by internalizing classifier-free guidance and eliminating paired conditional-unconditional passes.The distilled model is used throughout inference.
  • 4 Inference: Inference preprocesses reference video, audio, and scene prompts, generates a conditioned scene image, then performs DiT generation, super-resolution, and streaming VAE decoding.The pipeline combines all preprocessed signals through Sparse Reference Attention before one-step identity-aware upscaling.
  • 4.1 Chunk-Based Long-Form Generation: Each chunk produces 41 latent frames corresponding to 161 pixel frames at 25 fps, or approximately 6.4 seconds, supporting arbitrarily long temporally coherent videos.Chunks are connected through prefix conditioning.
  • 4.1 Chunk-Based Long-Form Generation: The first chunk uses ref2v conditioning, while later chunks use previous-frame prefixes with a 2-frame overlap for smooth, seamless transitions.This sequential strategy avoids separate interpolation chunks.
  • 4.1 Chunk-Based Long-Form Generation: A global appearance anchor from the first chunk, combined with inter-chunk motion propagation, preserves subject identity across arbitrarily long videos.The anchor is used for multi-chunk generation.
  • 4.2 Diffusion Sampling: Improved Stochastic Euler overshoots each flow-matching step and stochastically renoises it with fresh Gaussian noise, stabilizing high-frequency details at reduced step counts.The method targets hand clarity, dental structures, and fine textures.
  • 4.3 VideoRef Context Caching and Sparse Attention: Video-reference tokens remain invariant across denoising, enabling context-level caching at t = 0 and reuse across all subsequent denoising steps.The cached context includes latents, audio features, validity masks, expression embeddings, and identity embeddings, avoiding re-encoding across 24 steps.
  • 4.3 VideoRef Context Caching and Sparse Attention: Attention-level KV caching reuses reference projections, while sparse validity masks skip invalid tokens across self-attention, cross-attention, and FFN layers.Cached key-value tensors are concatenated with freshly computed generation-token tensors, and masks are precomputed per rank.

4.4 Distributed Inference with Sequence Parallelism · 4.5 Inference Acceleration

Avatar V distributes long-sequence inference across 8 GPUs using Ulysses Sequence Parallelism and FSDP2 CPU offloading. Its acceleration stack combines custom kernel synthesis, NVSHMEM pipelining, and system-level tuning, reducing latency through fused computation and stabilized distributed execution.

  • 4.4 Distributed Inference with Sequence Parallelism: Avatar V partitions video latents, reference tokens, face tokens, and conditioning tokens across 8 GPUs using Ulysses Sequence Parallelism with all-to-all attention communication.FSDP2 shards parameters with CPU offloading, freeing GPU memory for DiT activations and enabling rapid multi-model co-location through forward prefetching.
  • 4.5 Inference Acceleration: Production deployment targets kernel overhead, inter-GPU synchronization, and frequency-induced stragglers with custom compilation, NVSHMEM sequence parallelism, and system-level optimization.These bottlenecks arise from thousands of small operators, coarse-grained attention synchronization, and GPU-rank frequency variance at collective boundaries.
  • 4.5.1 Custom Compiler with Agentic Kernel Synthesis: torch.compile with Inductor is insufficient because it misses cross-operator fusion, generates suboptimal Triton kernels, and handles dynamic shapes poorly for production diffusion inference.The limitations leave memory bandwidth and model-specific tensor-shape optimizations unrealized.
  • 4.5.1 Custom Compiler with Agentic Kernel Synthesis: Engineers define fusion scopes, then an LLM-based agent generates CUDA/Triton candidates through iterative evolution for each specified operator fusion.The workflow combines human profiling expertise with automated kernel generation.
  • 4.5.1 Custom Compiler with Agentic Kernel Synthesis: 2–3 evolution islands explore 4 candidates per generation, selecting on kernel latency and numerical accuracy after a fixed iteration budget.Parallel islands mitigate noisy profiling caused by thermal state, allocator behavior, and scheduling variance.
  • 4.5.1 Custom Compiler with Agentic Kernel Synthesis: 3× latency reduction over the unoptimized baseline and 33% improvement over torch.compile result from mega kernels that fuse non-attention transformer-block operations.The compiled pass reduces thousands of small kernels to Flash Attention, cuBLAS GEMMs, and a handful of fused mega kernels.
  • 4.5.2 NVSHMEM-Based Sequence Parallelism: NVSHMEM replaces NCCL’s monolithic all-to-all operations across 8 GPUs with NVLink-direct, tile-level dataflow control inside fused kernels.Sub-tensor pipelining overlaps all-to-all scatter, cuBLAS GEMM computation, and all-to-all gather as tiles arrive.
  • 4.5.2 NVSHMEM-Based Sequence Parallelism · 4.5.3 System-Level Optimization: Up to 1.5× slowdown can result from naive SM partitioning, so profiling-guided auto-tuning balances NVLink bandwidth utilization against GEMM wave efficiency.NUMA-aware placement aligns ranks with local CPU cores and memory controllers, while GPU clock locking reduces block latency by approximately 3%.

4.6 Super-Resolution · 4.7 Streaming VAE Decode

Sections 4.6–4.7 combine detail-focused super-resolution with a streaming decode pipeline for high-resolution, lip-sync-sensitive output and bounded-memory video generation. The system enhances mouth-region detail while decoding and encoding incrementally, allowing early frame availability regardless of video length.

  • 4.6 Super-Resolution: A single-step adversarial super-resolution model converts low-resolution latent video into final high-resolution output.The model concentrates computation on high-detail regions.
  • 4.6 Super-Resolution: The super-resolution model particularly targets the mouth area to improve lip-sync fidelity.This focus directs computational resources toward perceptually important facial detail.
  • 4.6 Super-Resolution: σ = 0.6 noise is applied to low-resolution latents before super-resolution, balancing detail enhancement with structural preservation.The noise level provides room for enhancement while retaining the base generation’s structure.
  • 4.7 Streaming VAE Decode: The VAE decoder uses causal 3D convolutions and temporal feature caching for chunk-by-chunk video decoding.This avoids requiring the full video to remain in memory during decoding.
  • 4.7 Streaming VAE Decode: Decoded frames flow directly into an asynchronous streaming video encoder that writes the output file incrementally.Encoding proceeds as frames are decoded rather than waiting for the complete video.
  • 4.7 Streaming VAE Decode: The streaming pipeline maintains bounded memory consumption regardless of video length.Its chunked architecture prevents memory requirements from growing with the complete decoded sequence.
  • 4.7 Streaming VAE Decode: The pipeline makes the first frames available before the full video has been decoded.Asynchronous encoding enables output playback or access during ongoing decoding.

5 Data Curation

Avatar V curates large-scale pretraining and avatar-specific audio-to-video data through a shared multi-stage pipeline, branching from 50M raw videos into 100M+ pretraining clips and 10M+ fine-tuning clips. The curation combines automated quality and semantic analysis with human annotation and cross-clip identity connectivity for learning robust talking-head generation.

  • Corpus construction: The unified distributed pipeline orchestrates 25+ processing stages and 20+ specialized AI models across heterogeneous CPU and GPU infrastructure.It supports separate general human-centric pretraining and densely annotated avatar-specific fine-tuning regimes.
  • Automated curation: Smart clipping jointly maximizes clip duration subject to visual quality, motion, and face-presence constraints, while VLM filtering rejects scene changes, screencasts, games, and static photos.Clips are also categorized across 15 semantic dimensions for distribution balancing.
  • Automated curation: GPU nearest-neighbor indexing clusters near-duplicate clips and retains only the highest-quality clip from each cluster before 13 parallel stages extract visual, audio, caption, and embedding features.Extracted signals include lip-sync, pose, language, diarization, ASR word-level timestamps, and rich audio-video descriptions.
  • Avatar-specific curation: A2V fine-tuning clips receive ten avatar-specific quality signals covering gaze, blinking, facial clarity, teeth and hands, mouth openness, camera shake, lighting, and secondary speakers.These composable signals support flexible quality-tier construction without rerunning inference.
  • Corpus construction: 50M raw videos branch into 100M+ pretraining clips and 10M+ A2V fine-tuning clips after shared segment-level curation.Human annotation and cross-clip identity connectivity feed the avatar-specific branch.
  • Avatar-specific curation: Cross-clip connectivity links clips of the same individual across visually distinct scenes using high face similarity, low background similarity, and sufficient duration for dynamic-feature learning.This pairing helps disentangle identity from background, lighting, and pose.

6 Infrastructure

Avatar V’s infrastructure unifies heterogeneous multi-cloud GPU resources through HELIOS and replaces Ray with a declarative, purpose-built data-processing engine. The resulting platform supports large-scale, fault-tolerant scheduling with high utilization, rapid failure recovery, and linear scalability.

  • HELIOS: HELIOS unifies more than 5,000 GPUs across 5+ providers, 10+ regions, and 15+ standardized cells under reserved, on-demand, and preemptible capacity.It provides one platform for multi-cloud and large-scale operations.
  • HELIOS: HELIOS uses standardized cells, two-stage QoS-aware scheduling, and continuous resource governance to isolate failures and allocate heterogeneous GPU capacity by workload priority.Cells align with provider and region, while scheduling distinguishes inference, training, and data-processing requirements.
  • Data Processing Engine: At over 100K concurrent tasks and more than 2k nodes, Ray’s centralized GCS became a bottleneck, consuming over 100 GB RSS and 400% CPU.The GCS broadcast state changes to every node, producing traffic that scales quadratically with cluster size.
  • Data Processing Engine: The replacement data-processing engine declares desired state in a distributed key-value store, with nodes converging through an observe →diff →reconcile loop.This declarative design replaces imperative dispatch, acknowledgments, retries, and rollback complexity.
  • System Results: GPU utilization above 95%, node failure detection in under 30 seconds, and support for 5,000+ GPU nodes and 200K+ concurrent tasks demonstrate the engine’s operational scale.The platform also preserves running work through control-plane failures and enables zero-downtime deployments.

7 Evaluation

Avatar V is evaluated on a 70-case cross-scene talking-video benchmark using automated metrics and blinded human judgments across identity, synchronization, motion, artifacts, and visual quality. It achieves leading automated and subjective results, with strong identity transfer across scenes and frequent human-level realism despite a remaining detection gap.

  • Evaluation setup: The evaluation compares Avatar V with four state-of-the-art systems on a diverse cross-scene benchmark testing generalization beyond training scenes.The competitors are Kling O3 Pro, Veo 3.1, OmniHuman 1.5, and Seedance 2.0.
  • Evaluation setup: The benchmark contains 70 talking-video cases, pairing a reference clip with a different-scene clip whose first frame and audio drive generation.It tests identity transfer when reference and target videos depict the same individual in different scenes.
  • Qualitative evaluation: Avatar V maintains faithful identity transfer across visually distinct contexts and produces temporally coherent, naturally expressive outputs in the most challenging generated-scene condition.Generated scenes combine identity transfer with a novel, unseen background, while competing methods often lose identity consistency or introduce artifacts.
  • Automated metrics: 8.97 Sync-C and 6.75 Sync-D give Avatar V the strongest lip synchronization, while Face Similarity reaches 0.840 and Q-Align ties Seedance 2.0 at 4.85.Avatar V surpasses ground truth on Sync-C and Sync-D, while Veo 3.1 attains higher Q-Align quality at 4.95 but has Face Sim = 0.714.
  • Human evaluation: Avatar V achieves the highest MOS on all six perceptual dimensions, including 4.98 for identity consistency, 4.48 for motion naturalness, and 4.57 for motion consistency.The motion scores are attributed to the dedicated motion representation capturing individual-specific behavioral patterns.
  • Human evaluation: Avatar V wins against all four competitors with win rates from 68.9% to 85.7%, while annotators select the real video 77.8% overall and mistake generated video for real in 61.1% of cases.Annotator feedback highlights generation stability and holistic identity consistency, including appearance, behavioral patterns, and talking style.

8 Related Work

Recent video-generation research spans scalable diffusion architectures, audio- and image-driven portrait animation, video-reference conditioning, pose guidance, identity preservation, diffusion distillation, and preference alignment. These approaches establish the technical context for Avatar V while exposing tradeoffs in reference detail, efficiency, and controllability.

  • Diffusion-based video generation: Diffusion video generation progressed from Sora’s DiT-based minute-long videos to large-scale models such as CogVideoX, HunyuanVideo, Wan, Movie Gen, Kling, Lumiere, Step-Video, and SkyReels-V2.DiT has largely replaced U-Net, while Flow Matching has emerged as a practical alternative.
  • Audio-driven talking head generation: Audio-driven methods evolved from SadTalker’s 3DMM motion mapping to direct audio-to-video generation and hierarchical control of lip motion, expression, and pose.EMO removed intermediate representations, while Hallo and Hallo2 extended conditioning, duration, and resolution capabilities.
  • Single-image portrait animation: Single-image portrait animation methods improve efficiency and control through implicit keypoints, facial landmarks, hierarchical motion attention, and latent-space inpainting, but lack dynamic reference information.Examples include LivePortrait, AniPortrait, X-Portrait, and MuseTalk.
  • Video-reference-based approaches: Video-reference approaches include WanAnimate, SlotID, and Seedance 2.0, but bottleneck encoders can discard fine-grained details while concatenating all reference tokens incurs quadratic complexity.These methods condition generation on video references rather than single images.
  • Related enabling techniques: Reference-based human animation builds on pose guidance, identity-preservation modules, diffusion distillation, and reinforcement learning methods including Animate Anyone, IP-Adapter, Progressive Distillation, DPO, and RLHF for diffusion.These lines of work address temporal consistency, identity control, inference cost, and alignment with human preferences.

9 Ethics and Safety

The platform addresses avatar-generation ethics and safety through consent protections and a two-stage moderation pipeline. Custom-avatar creation requires verification, while represented individuals can request removal of their likeness.

  • Custom-avatar creation requires explicit verification from the represented individual.
  • Represented individuals retain the right to request removal of their likeness at any time.
  • Uploaded and generated content passes through a two-stage moderation pipeline combining automated review.

10 Conclusion

Avatar V generates high-fidelity talking avatar videos from short video references by conditioning directly on reference-video tokens, capturing both appearance and behavioral identity. Its supporting model, data, training, and deployment techniques enable state-of-the-art cross-scene performance across identity, synchronization, expression, motion, and visual-fidelity dimensions.

  • Avatar V generates high-fidelity talking avatar videos from short video references.
  • Sparse Reference Attention lets Avatar V attend directly to the full token sequence of a reference video for identity conditioning.
  • The system captures facial geometry, skin texture, accessories, talking rhythm, habitual expressions, and gestural tendencies without per-identity fine-tuning or bottleneck encoders.
  • Supporting techniques include a motion representation stream for closed-loop person-specific talking-style supervision and an identity-aware super-resolution refiner with sparse temporal attention.
  • Avatar V achieves state-of-the-art performance across identity preservation, lip synchronization, expression naturalness, motion quality, and visual fidelity on a cross-scene benchmark.The results are measured using automated metrics and comprehensive human evaluation.

Contributions and Acknowledgments

The paper acknowledges contributors listed in alphabetical order by first name, spanning the individuals named below.

  • Contributors are listed alphabetically by first name.
  • Named contributors include Benjamin Liang Ce Chen, Desmond Lin, Ivan Somov, Jiajun Zhao, Jiewei Yuan, Jingfeng Zhang, and Junhao Huang.
  • Additional named contributors include Nik Nolte, Pedram Haqiqi, Penghan Wang, Rong Yan, Rui Zhang, Sam Prokopchuk, Sivan Wang, and Viktor Goriachko.
Loading 2606.13872v1…