Source-linked AI summary
Kling-Omni Technical Report
Kling Team, Jialu Chen, Yuanzheng Ci, Xiangyu Du, Zipeng Feng, Kun Gai, Sainan Guo, Feng Han, Jingbin He, Kang He, Xiao Hu, Xiaohua Hu, Boyuan Jiang, Fangyuan Kong, Hang Li, Jie Li, Qingyu Li, Shen Li, Xiaohan Li, Yan Li, Jiajun Liang, Borui Liao, Yiqiao Liao, Weihong Lin, Quande Liu, Xiaokun Liu, Yilun Liu, Yuliang Liu, Shun Lu, Hangyu Mao, Yunyao Mao, Haodong Ouyang, Wenyu Qin, Wanqi Shi, Xiaoyu Shi, Lianghao Su, Haozhi Sun, Peiqin Sun, Pengfei Wan, Chao Wang, Chenyu Wang, Meng Wang, Qiulin Wang, Runqi Wang, Xintao Wang, Xuebo Wang, Zekun Wang, Min Wei, Tiancheng Wen, Guohao Wu, Xiaoshi Wu, Zhenhua Wu, Da Xie, Yingtong Xiong, Yulong Xu, Sile Yang, Zikang Yang, Weicai Ye, Ziyang Yuan, Shenglong Zhang, Shuaiyu Zhang, Yuanxing Zhang, Yufan Zhang, Wenzheng Zhao, Ruiliang Zhou, Yan Zhou, Guosheng Zhu, Yongjie Zhu
TL;DR
Kling-Omni addresses the fragmentation of video generation, editing, and multimodal reasoning by building a unified system for diverse visual-language inputs. It combines MVL interaction, an end-to-end architecture, multimodal training, and inference optimization, with evaluations reporting strong complex-task performance and state-of-the-art results. Some described features are not yet supported in the online version.
Problem
Existing video systems remain fragmented, while text-only control and limited native reasoning constrain precise multimodal video creation.
Method
Kling-Omni combines MVL inputs with a unified architecture, multimodal training, and optimized infrastructure for video generation, editing, and reasoning.
Results
Kling-Omni achieves state-of-the-art performance in complex tasks and shows superiority across evaluated dimensions for image-referencing and video-editing comparisons.
Takeaways & Limitations
Kling-Omni advances a single multimodal system that bridges video generation, editing, and multimodal reasoning.
Takeaways & Limitations
Some described features are not yet supported in the online version.
Abstract
from arXiv · showhide
We present Kling-Omni, a generalist generative framework designed to synthesize high-fidelity videos directly from multimodal visual language inputs. Adopting an end-to-end perspective, Kling-Omni bridges the functional separation among diverse video generation, editing, and intelligent reasoning tasks, integrating them into a holistic system. Unlike disjointed pipeline approaches, Kling-Omni supports a diverse range of user inputs, including text instructions, reference images, and video contexts, processing them into a unified multimodal representation to deliver cinematic-quality and highly-intelligent video content creation. To support these capabilities, we constructed a comprehensive data system that serves as the foundation for multimodal video creation. The framework is further empowered by efficient large-scale pre-training strategies and infrastructure optimizations for inference. Comprehensive evaluations reveal that Kling-Omni demonstrates exceptional capabilities in in-context generation, reasoning-based editing, and multimodal instruction following. Moving beyond a content creation tool, we believe Kling-Omni is a pivotal advancement toward multimodal world simulators capable of perceiving, reasoning, generating and interacting with the dynamic and complex worlds.
1 Introduction
Kling-Omni addresses fragmented video generation, editing, and understanding by unifying multimodal inputs and intelligent creation in one generalist framework. Its MVL interaction paradigm combines language and visual signals to improve control and support richer instruction understanding.
- Motivation and contribution: Text alone can underspecify spatial relationships, visual references, and temporal dynamics, creating a gap between user intent and generated output.
- Motivation and contribution: Existing video models often generate pixels effectively but struggle with semantic reasoning, scene logic, and complex user intentions.
- Motivation and contribution: Kling-Omni unifies video generation, editing, and intelligent creation tasks that are often handled by specialized models or separate pipelines.The framework is presented as a unified system that removes task boundaries.
- Motivation and contribution: MVL combines natural language with multimodal descriptions in a unified representation, treating text and visual signals as a cohesive interaction language.
- Motivation and contribution: Kling-Omni aims to move beyond rote generation by inferring user intentions and demonstrating reasoning capabilities.
2 Methodology
Kling-Omni uses an end-to-end architecture that accepts text, images, and video through a unified interface for multimodal video generation and editing. Its components enhance prompts, process cross-modal tokens, and refine generated content.
- 2.1 Model Architecture Overview: Kling-Omni processes text instructions, reference images, and video contexts through a unified interface for cinematic video creation and editing.
- 2.1 Model Architecture Overview: The Prompt Enhancer uses an MLLM to interpret heterogeneous inputs, infer creative intent, and reformulate prompts with learned world knowledge.
- 2.1 Model Architecture Overview: The Omni-Generator operates on visual and textual tokens in a shared embedding space to support cross-modal interaction, consistency, and instruction adherence.
- 2.1 Model Architecture Overview: A Multimodal Super-Resolution module refines the generated content within the framework’s end-to-end generation process.
2.2 Training Strategies of Omni-Generator
The Omni-Generator is trained through progressively richer multimodal data, preference optimization, and two-stage distillation. These strategies target instruction understanding, visual quality, and substantially cheaper inference.
- 2.2 Training Strategies of Omni-Generator: Pre-training uses large-scale text-video pairs and varied caption styles to establish instruction-based text-to-video generation across diverse instructional formats.
- 2.2 Training Strategies of Omni-Generator: Continue-training aligns the model with reference-to-video generation, image and video editing, and interleaved multimodal semantic-understanding tasks.
- 2.2 Training Strategies of Omni-Generator: Quality-tuning uses a high-quality, balanced dataset with precise instructions to optimize visual quality and multimodal understanding.
- 2.2 Training Strategies of Omni-Generator: DPO uses human-preference pairs focused on motion dynamics and visual integrity, avoiding the trajectory sampling cost associated with GRPO.
- 2.2 Training Strategies of Omni-Generator: Inference distillation compresses video synthesis from 150 NFE to 10 NFE while preserving output fidelity.The acceleration pipeline combines trajectory matching and distribution matching distillation.
- 2.2 Training Strategies of Omni-Generator: The second distillation stage combines distribution matching with preserved trajectory matching and trains few-step ODE sampling for the target tasks.
2.3 Prompt Enhancer
The Prompt Enhancer maps ambiguous multimodal prompts toward the model’s training distribution before generation. It uses a specialized MLLM dataset and two-stage SFT/RL training to improve reasoning and prompt quality.
- 2.3 Prompt Enhancer: The Prompt Enhancer maps diverse user prompts toward the training distribution to improve identity preservation, spatial coherence, color fidelity, and physical plausibility.
- 2.3 Prompt Enhancer: An MLLM-based Prompt Enhancer accommodates multimodal inputs through a specialized multilingual dataset tailored to generation tasks.
- 2.3 Prompt Enhancer: SFT first develops the model’s reasoning chain, followed by RL optimizing factual correctness, content richness, semantic plausibility, and prompt-data similarity.
2.4 Multimodal Super-Resolution
Kling-Omni’s multimodal super-resolution module combines low-resolution latents with multimodal vision-language signals and uses efficient attention and caching to produce detailed videos with faster inference.
- Multimodal Super-Resolution: The VSR model combines base-model low-resolution latents with multimodal vision-language signals in a unified framework for fine-grained video detail synthesis.The design targets high-fidelity visual details and textures across diverse applications.
- Multimodal Super-Resolution: Local window attention exploits video spatio-temporal redundancy, while shifted windows in odd layers preserve information flow between neighboring windows.The shifted-window strategy offsets windows by half their size to reduce receptive-field isolation.
- Multimodal Super-Resolution: Asymmetric attention restricts condition tokens to self-attention while noisy tokens attend to the full sequence, enabling condition-token KV caching across sampling steps.The cached features are reused in subsequent sampling steps.
- Multimodal Super-Resolution: Even- and odd-layer attention maps identify shaded regions whose computation can be skipped, reducing computational load for accelerated inference with a KV cache.Figure 2 distinguishes attention patterns between even-numbered and odd-numbered layers.
- Multimodal Super-Resolution: KV-feature reuse boosts generation efficiency with negligible impact on visual performance.
2.5 Training Optimization
Kling-Omni’s training system combines dynamic scheduling, multimodal sequence packing, memory-saving execution, and fault-tolerant infrastructure to improve utilization and recovery.
- Training Optimization: A central scheduler balances variable-length multimodal workloads across data- and pipeline-parallel groups during online VAE/text-encoder inference and DiT training.Tokens are dynamically partitioned across pipeline stages to balance encoding workloads and improve utilization.
- Training Optimization: Microbatch-level elastic Ulysses parallelism dynamically adjusts parallel degree and data-rank assignments, while two-tier all-to-all communication reduces cross-node congestion.An asynchronous scheduler predetermines the Ulysses degree for each microbatch.
- Training Optimization: Packed multimodal FlashAttention processes arbitrary cross-modal masks and variable-length sequences in one kernel, minimizing padding and layout-transform overhead.The computation graph preserves modality-independent computation while reducing redundant data movement.
- Training Optimization: Selective recomputation, pipeline-aware offloading, kernel fusion, and activation reuse reduce memory traffic, GPU memory, and computation in multi-view and multi-stream settings.
- Training Optimization: 97% Effective Training Time Ratio is achieved by compressing recovery time through automated fault detection, rapid restarts, and overlapped warmup operations.The system detects hangs within a minute, enables sub-minute restarts, and reduces first-iteration overhead to second-level.
- Training Optimization: Sequentialized dataset reads, asynchronous checkpointing, hardware isolation, and unified observability improve runtime stability and root-cause analysis.
2.6 Inference Optimization
Kling-Omni reduces inference cost through hybrid parallelism, near-lossless FP8 quantization, communication overlap, and caching tailored to long multimodal conditioning inputs.
- Inference Optimization: Hybrid Ulysses and tensor parallelism addresses GPU memory consumption and latency for long-sequence video generation, while overlap hides most communication cost.
- Inference Optimization: A hybrid quantization scheme provides nearly lossless acceleration by applying FP8 to most GEMM operations and self-attention modules.
- Inference Optimization: Fused quantization and dequantization operators minimize quantization overhead, while FP8 communication further reduces traffic and works with communication overlap.
- Inference Optimization: The tailored cache scheme achieves roughly a 2× speedup for long reference-image and reference-video inputs, with cache offloading alleviating memory pressure.
3 Data System
Kling-Omni’s data system organizes multimodal, cross-task training data through real and synthetic collection plus quality controls for temporal stability and cross-modal alignment.
- Data System: The data infrastructure spans image, text, and video modalities across generation, editing, and reference-based tasks to support temporal consistency and alignment.Figure 5 summarizes the constructed system’s cross-modal and cross-task distribution.
- Data System: The collection system combines broad real-world data acquisition with task-oriented synthetic construction to improve diversity, consistency, and controllability.
- Data System: Real-world videos and images provide broad scenario, subject, scene, and style coverage, while automated internet mining constructs semantically related cross-modal samples.
- Data System: Expert image-editing and video-understanding models generate synthetic editing and multi-image-reference samples, including automatically reverse-synthesized high-fidelity video data.
- Data System: A three-tier processing system governs basic quality, temporal stability, and cross-modal alignment so training data remains clean, stable, and interpretable.
- Data System: Quality governance filters invalid resolution or duration, duplicate content, audio-visual corruption, blur, jitter, compression noise, discontinuities, and misaligned multimodal content.
4 Model Performance
Kling-Omni is evaluated through a broad benchmark and human comparison covering reference-based generation, video editing, multimodal control, and reasoning-enhanced generation. The reported results show superiority over leading competitors and support diverse, compositional capabilities.
- 4.1 Evaluation Setup: The OmniVideo-1.0 Benchmark uses over 500 cases spanning multimodal reference, element integration, and video editing scenarios.The evaluation includes images, subjects, and videos as reference elements.
- 4.1 Evaluation Results: Kling-Omni demonstrates varying degrees of superiority across all evaluated dimensions against Veo 3.1 for image referencing and Runway-Aleph for video editing.The comparison uses aggregated Good–Same–Bad distributions from the OmniVideo-Benchmark 1.0.
- 4.2 Core Capabilities: Kling-Omni supports multi-modal, multi-dimensional references for identity, status, style, shot composition, and actions.Its subject library combines multiple views, poses, expressions, or lighting conditions to improve identity preservation.
- 4.2 Core Capabilities: The model supports arbitrary interactive edits across elements, styles, scenes, and shots, beyond addition, removal, and replacement.The reported examples cover editing operations from Fig. 17 to Fig. 23.
- 4.2 Core Capabilities: Kling-Omni composes multiple complex instructions in one generation process without sequential task execution or manual decomposition.The unified process is reported to reduce accumulated errors and improve generation efficiency.
- 4.3 Reasoning-Enhanced Generation: The system demonstrates visual-signal-guided generation and reasoning-enhanced tasks involving geometric sorting, relational inference, and visual-puzzle completion.These capabilities extend generation toward interpreting visual annotations and solving structured problems.
5 Conclusion
Kling-Omni bridges video generation, editing, and multimodal reasoning through a unified system for MVL inputs. Its data and infrastructure support high-fidelity video synthesis, while evaluations report state-of-the-art performance and a path toward multimodal world simulators.
- Conclusion: Kling-Omni bridges video generation, editing, and multimodal reasoning in a single holistic system.The model uses a shared embedding space aligned with a vision-language model.
- Conclusion: The system processes MVL inputs to produce high-fidelity, physically plausible video content.Its scope extends across multimodal visual-language interaction rather than isolated video-generation tasks.
- Conclusion: A comprehensive data engineering pipeline and optimized infrastructure support temporal stability, semantic alignment, scalability, and efficiency.These contributions extend beyond the model architecture.
- Conclusion: Extensive evaluations demonstrate state-of-the-art performance in complex tasks.This is stated as the report’s overall evaluation conclusion.
- Conclusion: Kling-Omni is positioned as a foundational step toward multimodal world simulators that perceive, reason, generate, and interact with dynamic worlds.The statement describes the report’s forward-looking scope.
6 Contributors
The supplied passages in this section consist primarily of contributor information and figure examples illustrating Kling-Omni’s supported generation, editing, reference, and reasoning scenarios.
- Reference-Based Generation: The section includes examples of image and video reference, camera and motion transfer, shot generation, and sketch-guided video creation.The figures cover reference-driven generation across multiple input forms.
- Reference-Based Generation: The examples include temporal narratives generated from multi-grid image inputs.Figures 15 and 16 illustrate this capability.
- Task Composition: Task-composition examples combine references, camera-angle changes, added elements, background replacement, and stylization in single workflows.Figures 24 and 25 illustrate combined instructions.
- Reasoning-Enhanced Generation: The reasoning examples include geospatial and temporal world knowledge, geometric sorting, and completion of intersecting Chinese idioms.These examples are presented as reasoning-enhanced generation.