KlingAvatar 2.0 Technical Report

• The expanded training dataset contains high-quality cinematic videos, multilingual and multi-character conversations, and filtering for visual fidelity and consistent audio-lip synchronization.It expands upon [13].Source: §1 · p. 3
• The spatial-temporal cascade progressively and in parallel upsamples low-resolution blueprint videos in time and space, improving efficiency and detail while mitigating temporal drift.The framework supports long-duration, high-resolution generation.Source: §1 · pp. 2–3
• A Co-Reasoning Director coordinates multiple MLLMs and LLMs through multi-turn dialogue to resolve modality conflicts and create shot-level storylines, while a Negative Director refines negative prompts.The negative director is intended to improve instruction following and character emotion expression.Source: §1 · pp. 2–3
• Deep DiT features support character-mask prediction and mask-controlled audio injection, enabling synchronized yet individually controlled animations from multiple audio streams.This mechanism targets complex multi-character conversational settings.Source: §1 · pp. 2–3
• KlingAvatar 2.0 reports state-of-the-art performance across visual quality, character animation, camera movement, lip synchronization, and audio-emotion alignment, with efficient, coherent videos up to 5 minutes.The reported comparisons include leading competitors [13] [30].Source: §1 · p. 3

Video Generation

• Video generation moved from temporalized U-Net diffusion [45] toward compressed-token DiT models [36] [38] [56] [67] [74], but still faces scalability, temporal-drift, and audio-conditioning limitations.Extensions address efficient long-context generation [7] [18] [32] [53] [70] [73], cascaded high-resolution synthesis [52], and world modeling [26] [51] [69]; general video methods remain inadequate for speech-driven digital humans.Source: §2 · p. 3

Multimodal Avatar Synthesis

• Avatar synthesis progressed from reference-video or landmark motion transfer [24] [42] [46] [47] [61] [72] and 3D methods [11] [25] to audio-driven generation.Audio-driven systems use explicit facial landmarks [8] [28] [57], implicit motion representations [19] [21] [71], or end-to-end cross-attention diffusion [29] [41] [55] [64] [65] [75]; semi-body methods synchronize gestures and upper-body movement [39] [54].Source: §2 · pp. 3–4
• Pretrained video diffusion improves avatar temporal coherence and visual fidelity [9] [10] [15] [16] [17] [35] [60], while extensions address human-object interaction [27] and multi-person audio control [33] [62] [63].These approaches support complex backgrounds, full-body interactions, and per-character audio control.Source: §2 · p. 4
• MLLM-based multimodal planning uses shot-level guidance for long-duration generation, enabling fine-grained expression, vivid emotions, reasoned actions, and environmental interaction [13] [30].This line of work extends avatar generation through multimodal planning.Source: §2 · p. 4

3.1 Spatial-Temporal Cascade Modeling

• A two-stage spatial-temporal cascade plans global video dynamics with low-resolution keyframes, then refines them into high-resolution, audio-synchronized clips with coherent transitions.High-resolution keyframes anchor clip expansion, while audio-aware interpolation and super-resolution improve temporal connectivity, lip synchronization, spatial consistency, and visual detail.Source: §3.1 · pp. 4–5

3.2 Co-Reasoning Director

• Three Co-Reasoning Director experts analyze audio, reference-image context, and text instructions through dialogue, resolving conflicts and producing storyline and segment-level plans.The Negative Director down-weights implausible poses, artifacts, opposing emotions, and motion styles; audio-aligned segment plans parameterize keyframe and clip refinement.Source: §3.2 · p. 5

3.3 Multi-Character Control

• For multi-character scenes, deep DiT features predict per-character masks that gate identity-specific audio injection to corresponding regions, following character-aware audio-injection work [33] [62] [63].An automated annotation pipeline uses YOLO detection, DWPose keypoints, and SAM2 segmentation and tracking, then filters misaligned masks for training.Source: §3.3 · pp. 5–6

3.4 Accelerated Video Generation

• For faster inference, KlingAvatar 2.0 selects trajectory-preserving distillation over PCM [58] and DCM [37] or distribution-matching DMD [68], with customized schedulers and multitask distillation.The schedulers balance speedup against performance, and the report states that multitask configurations improve outcomes across individual tasks.Source: §3.4 · p. 6

4.1 Experimental Setup

• Evaluation used 300 paired image, audio, and text cases—100 each of Chinese speech, English speech, and singing—with human GSB comparisons scored by (G+S)/(B+S).The protocol follows human-preference evaluation in [13]; higher scores indicate stronger human preference.Source: §4.1 · p. 7
• The extended GSB criteria assess face–lip synchronization, visual quality, motion plausibility and expressiveness, and text relevance.They cover synchronization and facial continuity, appearance and artifacts, coherent body/head/camera movement and tracking, expressive audio/text-matched gestures, and storyline, camera, and scene alignment with instructions.Source: §4.1 · pp. 7–8

4.2 Experimental Results

• KlingAvatar 2.0 shows strong overall GSB performance against HeyGen, Kling-Avatar [13], and OmniHuman-1.5 [30], with notable gains in motion expressiveness and text relevance.Quantitative results are summarized in Table 1 and visualized in Fig. 4.Source: §4.2 · p. 8
• KlingAvatar 2.0 produces more plausible hair motion and head poses, prompt-aligned camera movement, and expressive, audio-synchronized gestures and emotions.Comparisons describe baseline limitations in hair dynamics, camera instruction following, and folding hands in front of the chest.Source: §4.2 · p. 8
• Across diverse scenarios, the framework fuses image, audio, and text instructions into expressive characters, coherent body and camera motion, and precise lip synchronization, including multi-person interactions with per-character audio.These examples are presented as evidence of robustness and versatility in open, complex settings.Source: §4.2 · p. 8
• The negative director uses shot-specific prompts tied to storyline and target emotion, discouraging implausible expressions, unstable motion, and narrative-inconsistent artifacts.This contrasts with prior work's small, fixed set of generic negative prompts for an entire video.Source: §4.2 · p. 8

5 Conclusion

• Experiments demonstrate leading performance in visual fidelity, identity preservation, lip–audio synchronization, instruction-following, long-duration coherence, and multi-character, multi-audio controllability.The report identifies these as the framework's experimental strengths.Source: §5 · p. 8
• Multimodal experts reason across audio cues, visual contexts, and complex instructions in multi-turn dialogues to resolve conflicts and produce global storylines and local sub-clip prompts.Storylines guide low-resolution blueprint keyframes, while local prompts refine sub-clip dynamics.Source: §5 · p. 8
• The framework upscales blueprint keyframes into high-resolution, audio-synchronized sub-clips and composes long-form videos in parallel using first–last frame conditioning.Source: §5 · p. 8
• KlingAvatar 2.0 extends generation to multi-character settings with identity-specific audio control and develops an automated annotation pipeline for large-scale multi-person video datasets.Source: §5 · p. 8