PaperlayerView exact revision on arXiv

Source-linked AI summary

Kandinsky 5.0: A Family of Foundation Models for Image and Video Generation

Vladimir Arkhipkin, Vladimir Korviakov, Nikolai Gerasimenko, Denis Parkhomenko, Viacheslav Vasilev, Alexey Letunovskiy, Nikolai Vaulin, Maria Kovaleva, Ivan Kirillov, Lev Novitskiy, Denis Koposov, Nikita Kiselev, Alexander Varlamov, Dmitrii Mikhailov, Vladimir Polovnikov, Andrey Shutkin, Julia Agafonova, Ilya Vasiliev, Anastasiia Kargapoltseva, Anna Dmitrienko, Anastasia Maltseva, Anna Averchenkova, Olga Kim, Tatiana Nikulina, Denis Dimitrov

arXiv:2511.14993v3cs.CVcs.AIcs.LG

TL;DR

Efficiently generating high-quality, consistent, and controllable video remains a challenge, and automated evaluation metrics can miss attributes important to human judgment. Kandinsky 5.0 combines a model family, multi-stage training, and architectural optimizations; human side-by-side evaluations find its models superior or competitive with leading models across key metrics.

  • Problem

    Efficient generation of high-quality, consistent, and controllable video remains challenging, while automated metrics can correlate weakly with human judgment on complex attributes.

    Source: §1 · p. 4, §10.5 · p. 63

  • Method

    Kandinsky 5.0 combines a shared CrossDiT architecture with NABLA attention, curated image and video data, and multi-stage training and optimization.

    Source: §5.1 · p. 22, §13 · p. 67

  • Results

    Human side-by-side evaluations find Kandinsky 5.0 superior or competitive with leading models across visual quality, motion dynamics, and prompt adherence.

    Source: §13 · p. 67

  • Takeaways & Limitations

    The three model lineups are optimized for different generative tasks and efficiency requirements across high-resolution image and video generation.

    Source: §13 · p. 67

    Text-visual alignment slightly lags some competing solutions, which the authors attribute primarily to the text encoder’s 256-token context limit.

    Source: §11 · p. 64

Abstract

from arXiv · show

This report introduces Kandinsky 5.0, a family of state-of-the-art foundation models for high-resolution image and 10-second video synthesis. The framework comprises three core line-up of models: Kandinsky 5.0 Image Lite - a line-up of 6B parameter image generation models, Kandinsky 5.0 Video Lite - a fast and lightweight 2B parameter text-to-video and image-to-video models, and Kandinsky 5.0 Video Pro - 19B parameter models that achieves superior video generation quality. We provide a comprehensive review of the data curation lifecycle - including collection, processing, filtering and clustering - for the multi-stage training pipeline that involves extensive pre-training and incorporates quality-enhancement techniques such as self-supervised fine-tuning (SFT) and reinforcement learning (RL)-based post-training. We also present novel architectural, training, and inference optimizations that enable Kandinsky 5.0 to achieve high generation speeds and state-of-the-art performance across various tasks, as demonstrated by human evaluation. As a large-scale, publicly available generative framework, Kandinsky 5.0 leverages the full potential of its pre-training and subsequent stages to be adapted for a wide range of generative applications. We hope that this report, together with the release of our open-source code and training checkpoints, will substantially advance the development and accessibility of high-quality generative models for the research community.

1 Introduction

Kandinsky 5.0 targets the computational and quality challenges of video generation with a family of image and video models and a multi-stage training pipeline. Its CrossDiT attention optimization is reported to accelerate high-resolution, long-duration video generation while maintaining quality.

1 Introduction

  • The lineup comprises 19B Video Pro for high-resolution clips up to 10 seconds, 2B Video Lite for clips up to 10 seconds, and 6B Image Lite for high-resolution text-to-image and image editing.Video Pro and Video Lite support text-to-video and image-to-video generation.Source: §1 · p. 4
  • The report describes data collection and processing, including preparation for instructive image-editing tuning and self-supervised fine-tuning for video and image.Source: §1 · p. 5
  • Training combines pretraining and supervised fine-tuning with an RLHF adversarial method that compares generated images with images from the SFT dataset.The authors report improved realism, visual quality, and prompt alignment.Source: §1 · p. 5
  • NABLA reduces training and inference time by 2.7× at 90% sparsity for high-resolution video exceeding 512 px and longer than 5 seconds, while maintaining generated video quality.Quality is confirmed by FVD, VBench, CLIP-score, and human side-by-side evaluation.Source: §1 · p. 5
  • The pipeline also uses VAE optimization, text-encoder quantization, and CrossDiT training techniques; video distillation combines three approaches to reduce NFE from 100 to 16 while preserving visual quality.Final models are evaluated against state-of-the-art approaches, and code and weights from multiple training stages are released through diffusers.Source: §1 · p. 5

2 Report Overview

The report overview maps the paper from model history and data processing through architecture, training, and optimization. It then outlines evaluation, use cases, related work, limitations, ethical considerations, and conclusion.

Report Overview

  • Sections 3–4 trace the Kandinsky family and describe data curation and annotation for image and video pretraining, fine-tuning, editing, and Russian-specific multicultural data.Source: §2 · p. 6
  • Sections 5–7 cover the shared CrossDiT architecture and NABLA attention, multi-phase training, and optimization of VAE encoding, CrossDiT training, and GPU memory use.Source: §2 · p. 6
  • Sections 8–10 present visual-quality growth and human side-by-side evaluations, generation use cases, and related work across generative models, post-training, and evaluation.Source: §2 · p. 6
  • Sections 11–14 address remaining challenges and future research, responsible-AI practices, and the conclusion, contributors, and acknowledgments.Source: §2 · p. 6

3 Background: The Evolution of Kandinsky models

The Kandinsky family evolved from autoregressive image models to diffusion and then transformer-based image and video models. Kandinsky 5.0 is presented as the family’s first Flow Matching-based generation family, with six models for different tasks and sizes.

3 Background: The Evolution of Kandinsky models

  • Kandinsky 1.0, released in June 2022, was a 12B-parameter autoregressive image model trained on 120 million text-image pairs and generating 256×256 images.It enlarged the earlier 1.3B-parameter Malevich model.Source: §3 · p. 7
  • Kandinsky 2.0 introduced diffusion in November 2022, using two encoders for 101-language support and training on one billion text-image pairs.The 2B-parameter model generated 512×512 images; Kandinsky 2.1 added a single text encoder, MoVQ autoencoder, and image prior.Source: §3 · pp. 7–8
  • In 2023, Kandinsky 2.2 raised resolution to 1024 pixels and added ControlNet and four-second clips, while Kandinsky 3.0 emphasized text-image alignment and Kandinsky Video 1.0 generated eight-second videos in two stages.Kandinsky Video 1.0 generated an 8-second, 512×512 video in approximately three minutes.Source: §3 · p. 8
  • April 2024 brought four-step Kandinsky Flash and Kandinsky 3.1, which used Flash as a refiner; Kandinsky Video 1.1 followed with 5.5-second videos and 4.6 million text-video training pairs.Kandinsky 3.1 also improved multicultural awareness, particularly for Russian culture.Source: §3 · p. 8
  • Kandinsky 4.0 was the family’s first Diffusion Transformer model, generating images and 12-second videos; Kandinsky 4.1 later adopted a DiT image architecture and artist-selected SFT data.Kandinsky 5.0 is described as the family’s first Flow Matching-based model family, comprising six models for image and video generation.Source: §3 · p. 8

4 Data Processing Pipeline

This section describes the collection and processing of Kandinsky’s image, video, and image-editing datasets, including staged filtering, annotation, and curation to prepare training data. It also covers video-scene quality screening, synthetic captions and metadata, dataset organization and analysis, infrastructure, and specialized supervised fine-tuning data curation.

4.2 Text-to-Image Dataset Processing

  • Kandinsky T2I comprises more than 500 million general-domain images from open datasets and online repositories for pretraining and fine-tuning.Sources include prominent open datasets and large online image repositories.Source: §4.2 · p. 9

Processing Pipeline

  • The T2I pipeline removes low-resolution and duplicate images, then filters watermarks, excessive text, and overly simple content using quality and complexity assessments.Images with a shorter side below 256 pixels are discarded; complexity assessment combines segmentation masks with edge analysis.Source: Processing Pipeline · pp. 9–10
  • T2I processing annotates and classifies image content, then generates synthetic captions with multimodal models and post-processes them for consistency.Caption generation includes detailed English captions and, for higher-resolution images, Russian captions; non-English characters are filtered from English captions.Source: Processing Pipeline · p. 11

4.3 Image Editing Instruct Dataset Processing

  • The I2I instruct pipeline matches candidate images using semantic, visual, and face similarity, verifies geometry, filters unsuitable overlaps, and generates transformation-focused captions.Captioning models were compared by human side-by-side evaluation; GLM 4.5 was selected for its performance-to-cost ratio and fine-tuned with LoRA.Source: Processing Pipeline · pp. 12–13

Dataset Summary

  • Approximately 150 million curated image pairs with textual instructions form the Kandinsky I2I dataset for image-editing model training.The filtering and verification pipeline selects pairs that describe transformations; the dataset also supports a further supervised fine-tuning dataset.Source: Dataset Summary · p. 13

Supervised Fine-Tuning (SFT) Dataset Curation

  • The SFT dataset is manually curated from about 600,000 candidate pairs selected by target-image Q-Align quality and aesthetic thresholds.The thresholds are Q-Align score greater than 4 and aesthetic score greater than 2; annotators select the final high-quality pairs.Source: Supervised Fine-Tuning (SFT) Dataset… · p. 13

4.4 Text-to-Video Dataset Processing

  • Kandinsky T2V contains more than 250 million video scenes; its pipeline segments scenes, filters short-side resolutions below 256 pixels, and deduplicates with video perceptual hashes.Scenes are constrained to 2–60 seconds, and the hash-based method can leave some duplicates in the dataset.Source: §4.4 · p. 15, Processing Pipeline · p. 15

Processing Pipeline

  • Scenes are screened for extreme staticity or dynamism using MS-SSIM, visual quality with DOVER [81] and Q-Align [67], and excessive text with CRAFT [68].DOVER combines technical and aesthetic quality scores as a weighted sum; text detections are averaged across three evenly spaced frames.Source: Processing Pipeline · p. 16
  • Objects are detected in five frames per scene; a CLIP-based classifier [51] uses frame embeddings for location, style, main-subject, and detailed-place classification; an architecture predicts scores for camera/object movement and light/color dynamics.Source: Processing Pipeline · p. 16
  • Scenes that pass filtering receive synthetic English captions from Tarsier2-7B17, which are post-processed to remove introductory phrases and exclude sentences containing non-English characters.The language filter addresses occasional incorrect or mixed-language model output.Source: Processing Pipeline · p. 16
  • InternVideo2-1B [84] embeddings are clustered into 10,000 groups for balanced sampling and dataset analysis, while processed scenes and metadata are stored in resolution-grouped Parquet files on S3-compatible storage.The files include filter scores and captions and are grouped by shortest frame side: 256, 512, or 1024.Source: Processing Pipeline · p. 16

4.5 Russian Cultural Code Dataset Processing

This material covers Russian Cultural Code (RCC) data and SFT curation, then compares domain-organized SFT-soup strategies; hierarchical domain organization performs best in side-by-side evaluations.

4.5 Russian Cultural Code Dataset Processing

  • The RCC dataset contains 229,504 video scenes and 768,555 images focused on Russian cultural features.These features include faith, language, historical memory, nature, architecture, and personality; Figure 7 describes the curated T2V data distribution by location, main object, and scene style.Source: §4.5 · p. 16, Figure 7 · p. 17

Dataset Characteristics

  • RCC data is manually curated for cultural relevance and visual quality, with Russian descriptions machine-translated into English.The dataset is used for pretraining and specialized fine-tuning on culturally specific content.Source: Dataset Characteristics · p. 17

4.6 Supervised Fine-Tuning Dataset Processing

  • The SFT dataset combines expert-selected high-fidelity image and video examples, with nine-domain classification and finer image classes.Its organization supports domain-specific curricula, balanced sampling, specialized fine-tuning, controlled image-video mixing, and SFT-soup composition by weight averaging.Source: §4.6 · pp. 17–18

Dataset Construction

  • SFT data collection began with 93,296 video scenes and about 10 million images, then strict and relaxed thresholds produced two dataset versions.The strict version retained 2,833 video scenes and 45,000 images; the relaxed version retained 12,461 scenes and 153,000 images. Fine-tuning samples from both with calibrated probabilities.Source: Dataset Construction · pp. 18, 20
  • Image captions are generated in multiple lengths and both English and Russian, while video captioning produces cleaned variants of differing lengths.The image pipeline uses Qwen2.5-VL-32B-Instruct followed by Qwen3-32B; video captions use SkyCaptioner-V1 and Qwen2.5-VL 32B and 72B.Source: Dataset Construction · p. 18

VLM Domain Organization

  • A vision-language model classifies image and video data into nine consistent domains, and the T2V SFT-soup averages domain-model weights equally.The shared domains are intended to maintain coherence between image and video data during mixed-dataset T2V training.Source: VLM Domain Organization · p. 20

Advanced Domain Organization and SFT-soup Composition Ablation Study

  • For T2I, the study compares VLM domains, nine-way CLIP clustering, and hierarchical VLM domains split into 2–9 coherent subdomains.The hierarchical subsets are separately fine-tuned to target visual characteristics while maintaining overall coherence.Source: Advanced Domain Organization and… · p. 22
  • Hierarchical domain organization achieved the best side-by-side results, with high realism, improved text rendering, and compositionally correct images.Equal or square-root-proportional weights performed best among the three SFT-soup merging strategies. The resulting SFT dataset is described as high-aesthetic content for fine-tuning across visual domains.Source: Advanced Domain Organization and… · p. 22

5 Kandinsky 5.0 Architecture

Kandinsky 5.0 uses a unified latent-diffusion, Flow Matching architecture centered on CrossDiT. Its NABLA attention mechanism adapts sparse masks to video content and is reported to accelerate generation while maintaining quality.

5.1 Model Overview

  • All Kandinsky 5.0 models share a latent-diffusion architecture trained with Flow Matching, with a cross-attention Diffusion Transformer (CrossDiT) as its core.CrossDiT block counts vary by model size, and the family uses text and visual encoders.Source: §5.1 · p. 22

5.2 Diffusion Transformer (CrossDiT) Architecture

  • CrossDiT processes text embeddings, diffusion time, a CLIP description embedding, and image or video latents; refined text queries enter its blocks through cross-attention.The Linguistic Token Refiner enhances text representations and removes positional bias inherited from the pretrained text encoder.Source: §5.2 · p. 23

5.3 CrossDiT Block Architecture

  • CrossDiT blocks sequence self-attention, cross-attention, and an MLP; their cross-attention design supports sparse attention for variable-length videos without the concatenation used in Kandinsky 4.0.The authors report that eliminating concatenation avoids a significant training slowdown.Source: §5.3 · p. 24, Figure 11 · p. 24

5.4 Neighborhood Adaptive Block-Level Attention

  • NABLA pools queries and keys in groups of 64, selects relevant blocks by per-head CDF thresholding, and can union its mask with Sliding-Tile Attention to suppress border artifacts.Figure 12 depicts the mask-construction stages.Source: §5.4 · p. 25, Figure 12 · p. 25
  • NABLA also fractal-flattens tokens from 8×8 spatial patches into contiguous sequences, then reverses the ordering at the network output.This layout keeps spatially adjacent tokens contiguous in memory.Source: §5.4 · pp. 25–26, Figure 15 · p. 26
  • Up to 2.7× speedup in training and inference, with equivalent generation quality to full attention, is reported for NABLA.The quality assessment uses CLIP, VBench, and human evaluations.Source: §5.4 · p. 26

6 Training Stages

This section describes Kandinsky 5.0’s training stages and supporting strategies, from pre-training and supervised fine-tuning to video distillation and image-model RL post-training. It also details reward-model selection and KL-regularized DRaFT-K tuning, including the use of real SFT images as relative-reward references.

6.1 Training Infrastructure

  • Pre-encoded VAE latents streamed from S3 into aspect-ratio queues; HSDP partitioned DiT weights across 64 GPUs and text-encoder weights across 32, yielding low per-GPU model-parameter and optimizer-state memory.Adaptive attention made processing one long video comparable in cost to several shorter clips with the same total tokens, benefiting dynamic batching. Checkpointing wrote each parameter shard directly to storage without reconstructing the full state_dict.Source: §6.1.1 · p. 27, §6.1.2 · pp. 27–28, §6.1.3 · p. 28

6.1.3 Distributed Training and Memory Optimization

  • Asynchronous activation offloading substantially reduced peak activation memory without noticeably slowing training, and preserved a similar compute pattern to standard checkpointing.Pre-training used classical activation checkpointing; RL fine-tuning offloaded transformer-block inputs to host RAM between forward and backward passes.Source: §6.1.3 · p. 29

6.2 Training Procedure Overview

  • The training procedure combines shared image pre-training with model-specific high-resolution, editing, video, SFT, distillation, and RL stages.Image Lite shares low- and medium-resolution text-to-image pre-training; T2I Lite continues at high resolution with SFT and RL, while Image Editing adds medium- and high-resolution instructional editing and SFT. Video Lite trains text-to-image, text-to-video, and image-to-video at probabilities 1%, 79%, and 20%; Video Pro uses 2%, 77%, and 21% across four pre-training stages and SFT, increasing resolution to 1024 pixels and video length to 10 seconds.Source: §6.2 · pp. 29, 31

6.3.1 Regimes

  • CrossDiT training covers text-to-image, instruct editing, text-to-video, and image-to-video regimes, with NABLA used for 1024-resolution or up-to-10-second-at-512-resolution video generation.Image-to-video keeps the first frame unnoised and marks it with a one-valued mask; instruct editing concatenates a reference image and one-valued mask with the noisy input.Source: §6.3.1 · pp. 31–33

6.4.1 Image Generation

  • Weighted averaging of nine domain-specific image models improved all target metrics in side-by-side human evaluation, yielding high realism, text alignment, and compositional quality.Direct full-dataset fine-tuning maintained a non-photorealistic style and degraded text alignment and rendering; hierarchical clustering and subdomain fine-tuning were used instead.Source: §6.4.1 · p. 33

6.4.2 Video Generation

  • Video model souping outperformed standard SFT in internal evaluations for visual quality, motion consistency, and artifact avoidance while maintaining equivalent text alignment.The video SFT dataset contained approximately 2.8 thousand videos and 45 thousand images; nine domain-specific models were uniformly averaged, while individual models were prone to overfitting and artifacts.Source: §6.4.2 · p. 35

6.5 Distillation

  • CFG distillation reduced sampling from 100 to 50 NFEs, a 2x speedup, with comparable quality; internal human evaluation found no perceptible degradation in composition, detail, coherence, or alignment.A more aggressive two-stage diffusion distillation produced the basis for Video Lite Flash and Video Pro Flash; its 16-NFE first stage was followed by adversarial refinement to address reduced visual fidelity.Source: §6.5 · pp. 35–36

6.6 RL-based Post-training for Image Generation

  • The image RL post-training pipeline trains a reward model from ordered real, pre-trained, and SFT-generated image comparisons, then uses its feedback to fine-tune the image generator.Initialized from Qwen 2.5VL-7B, the model predicts whether the first of two annotated images is better; the probability of “Yes” supplies the reward.Source: §6.6 · p. 36, Reward Model Training · p. 37

Reward Model Training

  • The reward model checkpoint trained for 1300 steps was selected after later training showed increasing overfitting on validation scores.The team monitored the score distribution and Gaussian-kernel-density-estimation standard deviation σ to avoid overfitting.Source: Reward Model Training · p. 38

RL-based post-training

  • DRaFT-K tunes images by backpropagating through the final generation steps to maximize reward-model preference while regularizing against the SFT model with KL divergence.For text-to-image training, βKL = 2e-2 and K = 10; the KL term compares the trainable and frozen post-SFT models over the gradient-updated steps.Source: RL-based post-training · pp. 38–39
  • Among tested alternatives, relative reward using a real SFT-dataset image as the reference produced the best image quality.Experiments also considered differentiable absolute reward, two generated images for relative reward, and selecting the best of N generations as the reference.Source: RL-based post-training · p. 39

7 Optimizations

The optimization section describes faster VAE encoding, attention and inference optimizations, and analytical estimates for training time and GPU memory. Reported gains include faster encoding and diffusion-step caching, while NABLA is used for longer or higher-resolution generation.

7.1 VAE Encoder Acceleration

  • The Hunyuan Video VAE encoder achieved an average 2.5× encoding speed-up without additional training, and temporal blending of tiles reduced reconstruction quality.The authors also report that fewer tiles mean less quality loss.Source: §7.1 · p. 39

7.2 CrossDiT Optimization

  • CrossDiT optimization also uses profiling, inference-code refactoring, and torch.compile to reduce GPU idle time and improve efficiency.The section identifies the diffusion transformer as the primary computational bottleneck in image and video generation.Source: §7.2 · p. 39
  • 2.7× faster inference than full attention is reported for NABLA on generation over 5 seconds or at HD resolution; MagCache delivered 46% speedup with no visible quality drop.For standard-definition generation up to 5 seconds, the pipeline uses Flash Attention or Sage Attention depending on GPU hardware.Source: §7.2 · p. 40

7.3.1 Training Step Estimation.

  • The training-step estimate models duration from training and model configuration parameters, and the authors report that it fits measured step time and supports experiment planning.The estimates were used to select batch and parallel configurations.Source: §7.3 · p. 40, §7.3.1 · p. 40, §7.3.2 · p. 41

7.3.2 GPU Memory Consumption.

  • The GPU-memory estimate is presented as an analytical model of memory consumption; the authors report that the model fits measured memory use and captures scaling with model size, resolution, and hardware.The estimates supported experiment planning and configuration selection.Source: §7.3.2 · p. 41

8 Results

Human evaluation reports gains in visual quality and motion for Kandinsky 5.0 Video Lite against Wan models and Kandinsky 4.1 Video, alongside trade-offs in prompt following and speed. Results also cover Video Pro, Image Lite, image editing, and the Video Lite Flash variant.

8.1 Quality Progress

  • The section presents qualitative progress across image pretraining, SFT, and RL-based fine-tuning, and video pretraining and SFT; the Sora comparison uses pairwise judgments on MovieGen.Figure 28 describes approximately 65,000 judgments from 44 trained raters across 1,002 prompt-video pairs, with approximately 71% inter-rater agreement; the supplied passages do not state which model was preferred.Source: §8.1 · p. 41, Figure 26 · p. 42, Figure 27 · p. 43

8.2.4 Kandinsky 5.0 Video Lite vs. Wan Models

  • Kandinsky 5.0 Video Lite consistently leads Wan 2.1 14B, Wan 2.2 5B, and Wan 2.2 A14B in Visual Quality and Motion Dynamics, while Wan models lead in Prompt Following.The authors describe the prompt-adherence gap as moderate and variable by scenario, and identify a trade-off between semantic fidelity and visual fluency.Source: §8.2.4 · p. 45

8.2.5 Kandinsky 5.0 Video Lite vs. Kandinsky 4.1 Video

  • Kandinsky 5.0 Video Lite improves over Kandinsky 4.1 Video across metrics, particularly in motion dynamics, object realism, artifact suppression, and semantic accuracy.The comparison also reports strong face stability and component-level realism.Source: §8.2.5 · p. 48, Figure 31 · p. 48

8.2.6 Kandinsky 5.0 Video Pro vs Veo 3 and Veo 3 fast

  • Veo 3 and Veo 3 Fast outperform Kandinsky 5.0 Video Pro in Prompt Following, while Video Pro scores higher in Video Quality and Motion Dynamics.The authors identify precise prompt adherence as an area for further improvement.Source: §8.2.6 · p. 49

8.2.7 Kandinsky 5.0 Video Pro vs Wan 2.2 A14B

  • Kandinsky 5.0 Video Pro outperforms Wan 2.2 A14B in Visual Quality and Motion Dynamics in the reported comparison.The comparison covered text-to-video and image-to-video modes and also evaluated Prompt Following.Source: §8.2.7 · p. 49, Figure 33 · p. 50

8.2.8 Kandinsky 5.0 Image Lite and Image Editing

  • Kandinsky 5.0 Image Editing was compared with FLUX.1 Kontext [dev] and Qwen-Image-Edit-2509 on instruction following and visual quality; the results are presented in Figure 35.The comparison used image-instruction pairs from Kontext Bench.Source: §8.2.8 · p. 50
  • Kandinsky 5.0 Image Lite shows stronger Visual Quality while remaining competitive in Prompt Following against the evaluated image models.The internal comparison used a custom prompt set based on PartiPrompts and expanded with Giga Max.Source: §8.2.8 · pp. 49–50, Figure 34 · p. 51

8.2.9 Kandinsky 5.0 Video Lite Flash

  • Video Lite Flash shows a moderate performance drop, especially in fine detail, temporal coherence, and complex prompt semantics; lighter prompts and shorter clips show minimal perceptible difference.The authors frame this as a trade-off between reduced computational cost and faster inference versus output fidelity.Source: §8.2.9 · p. 52

9 Use cases

The use cases span text-to-image generation, instruction-driven image editing, text-to-video synthesis, and image-to-video animation. The passages describe example capabilities, operating formats, and prompt schemas for each mode.

9.1 Text-to-Image

  • Kandinsky 5.0 Image Lite supports diverse text-to-image outputs, including photorealism, traditional and digital art styles, logos, typography, and prompted legible text.Examples are shown in Figures 37 and 38.Source: §9.1 · p. 52

9.2 Image Editing

  • Kandinsky 5.0 Image Editing applies text instructions to modify existing images through operations such as object removal or insertion, attribute edits, style transfer, and sketch-to-image synthesis.The described process aims to preserve global coherence and local details; examples include inpainting, outpainting, and style conversion.Source: §9.2 · p. 59

9.3 Text-to-Video

  • Text-to-video outputs support 5- and 10-second durations at 24 fps; Video Lite supports up to 768 pixels on the longer side, while Video Pro supports up to 1408 pixels.The suggested prompt schema combines a main subject, action or motion, environment, and cinematic or camera parameters.Source: §9.3 · p. 59

9.4 Image-to-Video

  • Image-to-video generates dynamic sequences from a static image and text guide, with 5- and 10-second durations and 24 fps output.The suggested prompt schema focuses on primary motion, temporal characteristics, and camera movement.Source: §9.4 · p. 59

10 Related Work

The related-work discussion traces image and video generation approaches, attention and post-training optimizations, distillation, and evaluation. It emphasizes computational demands and the limits of automated metrics for assessing complex visual and temporal qualities.

10.1 Image Generation

  • Image generation progressed from GANs [110], VAEs [111], and autoregressive models [114] to diffusion [1], latent diffusion [4], and transformer-based DiT [17] approaches.The discussion associates these developments with trade-offs involving training stability, output quality, inference cost, and scaling.Source: §10.1 · p. 60

10.2 Video Generation

  • Video generation extends beyond text-to-video to image-, reference-, and video-to-video tasks, video-to-audio, frame interpolation, and camera control.Open-source efforts including HunyuanVideo, Mochi, CogVideoX [28], and Wan have narrowed, but not closed, the gap with proprietary systems.Source: §10.2 · pp. 60–61

10.2.1 Attention mechanism optimizations

  • Video attention scales quadratically with spatiotemporal latent size; cited optimizations include Flash Attention [124],[107] and hardware-aware Sliding Tile Attention.Flash Attention is reported to provide 2.4–3.1× speedups; the cited approaches use tiled computation and memory-oriented optimizations.Source: §10.2.1 · p. 61
  • An asynchronous memory pipeline achieves 58.79% MFU, while training-free attention-window configuration achieves 1.36× speedup with 98% quality retention.The training-free approach configures window size per attention head through score profiling.Source: §10.2.1 · p. 61

10.3 Post-training RL-based Techniques

  • RL-based alignment can use human- or VLM-annotated preferences, direct optimization such as DPO, or reward models; cited work reports comparative image-pair feedback as more robust than single-image feedback.The discussion distinguishes gradient-based reward optimization from RLHF methods that do not require gradients through reward-model outputs.Source: §10.3 · pp. 61–62

10.4 Distillation Methods

  • Diffusion distillation reduces network evaluations by learning a simpler trajectory; deterministic methods target teacher outputs, while distributional methods approximate output distributions.The review covers progressive, consistency, and adversarial distillation, as well as video-generation applications.Source: §10.4 · pp. 62–63

10.5 Generative Model Evaluation

  • Automated metrics can miss semantic coherence, motion realism, compositional reasoning, and temporal consistency; T2I-CompBench++ [148] and T2V-CompBench add specialized benchmarks and metrics.T2V-CompBench reports difficulty with dynamic attribute changes, motion direction, and multi-object counting.Source: §10.5 · p. 63
  • Pairwise side-by-side human evaluations offer interpretable attribute-level judgments and stronger alignment with perceptual quality than automated scores, but require resources and reliability safeguards.The discussion identifies trained annotators, sufficient rater overlap, and inter-rater agreement measurement as design considerations.Source: §10.5 · p. 63

11 Limitations and Further Work

The authors identify limitations in text alignment, long-range dynamics, generalization, model integration, and computational efficiency, with corresponding directions for further work.

Text-Visual Alignment

  • Prompt understanding lags some competing solutions; the authors attribute this primarily to the Qwen2.5-VL 7B text encoder’s 256-token context length.They propose exploring longer-context text encoders and RL-based fine-tuning.Source: §11 · p. 64

Temporal Consistency for Complex Dynamics

  • Long-range complex physical interactions, including fluid dynamics and cloth simulation, remain challenging; sequences longer than 10 seconds can occasionally show artifacts.The authors identify physical realism and long-term temporal consistency as targets for a future model iteration.Source: §11 · p. 64

Generalization Ability

  • Performance varies across styles, objects, and scenes, which the authors link to training-data class imbalance and stylistic or semantic biases.They are investigating more representative, higher-quality training data and intelligent curation.Source: §11 · p. 64

Creating a Foundation Visual Model

  • Kandinsky 5.0 is currently a family of specialized open-source models for specific generative tasks, rather than one unified multimedia foundation model.Consolidating capabilities into a single model is described as a long-term goal.Source: §11 · p. 64

Computational Efficiency

  • Real-time generation at 24+ FPS on consumer-grade hardware remains challenging at high resolutions, despite the reported optimizations.The authors are pursuing more efficient architectures and inference techniques for resource-constrained devices.Source: §11 · p. 64

12 Border Impacts and Ethical Considerations

Kandinsky 5.0 may inherit and amplify societal biases, reproduce stereotypes in response to some prompts, and misunderstand ambiguous instructions. The report also warns of misuse risks and places responsibility for ethical and legal use on users.

Open-Source Release

  • The MIT-licensed release omits built-in content filters, and users are liable for generated content and must use the model ethically and legally.Source: §12 · p. 64

Inherent Sociocultural Biases

  • As a web-scale-trained model, Kandinsky 5.0 can inherit and amplify societal biases.Source: §12 · p. 65
  • For prompts about profession, gender, age, or ethnicity, the model often reproduces common stereotypes, resulting in a lack of diversity and reinforcing biases found in popular culture.The passage encourages specific, descriptive prompts that explicitly call for diversity to achieve more balanced and representative outputs.Source: §12 · p. 66

Technical Limitations and Unintended Outputs

  • Ambiguous or incomplete prompts can lead to nonsensical, offensive, or otherwise undesirable content because the model lacks deep semantic language understanding.Source: §12 · p. 66

Potential for Misuse

  • Synthesizing realistic images and video carries misuse risk; the report prohibits disinformation, harassment, and illegal content, placing mitigation responsibility on end users.Source: §12 · p. 66

13 Conclusion

Kandinsky 5.0 combines image and video models with data, training, and architectural contributions aimed at high-quality, efficient generation. The report also identifies remaining limitations in text-visual alignment, long-term temporal modeling, and domain generalization.

13 Conclusion

  • The data pipeline targets high-quality, diverse, culturally aware image and video datasets, including specialized Russian cultural content and supervised fine-tuning data.Source: §13 · p. 67
  • The multi-stage training pipeline combines pre-training, supervised fine-tuning, distillation, and RL-based post-training to enhance visual quality, prompt alignment, and temporal consistency.Source: §13 · p. 67
  • CrossDiT and NABLA reduce computational complexity and accelerate training and inference, while pipeline optimizations target consumer and professional hardware.Source: §13 · p. 67
  • Human side-by-side comparisons show Kandinsky 5.0 models achieve superior or competitive results against Sora, Veo, and Wan across visual quality, motion dynamics, and prompt adherence.Source: §13 · p. 67
  • The Image Lite offering provides high-fidelity text-to-image generation and image editing with aesthetic and compositional control.Source: §13 · p. 67
  • Video models support text-to-video and image-to-video synthesis for clips up to 10 seconds at resolutions up to 1408p, with distilled variants that maintain quality while reducing inference time.Source: §13 · p. 67
  • The authors identify text-visual alignment, long-term temporal modeling, and generalization across visual domains as remaining limitations.Source: §13 · p. 67
  • The models, code, and training methodologies are released under the MIT license to support innovation, collaboration, and responsible use.Source: §13 · p. 67

Related papers

Loading 2511.14993v3…