PaperlayerView exact revision on arXiv

Source-linked AI summary

Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

Team Kandinsky, Julia Agafonova, Bulat Akhmatov, Mikhail Aksyutin, Grigorii Alekseenko, Anastasia Aliaskina, Olga Androsova, Vladimir Arkhipkin, Anna Averchenkova, Alexander Belykh, Serafima Bocharova, Sofiya Bogakovskaya, Anton Bukashkin, Mark Bulygin, Kirill Buzygin, Irina Cheremnykh, Kirill Chernyshev, Mikhail Chernyshov, Vladimir Chernyy, David Chikovani, Georgy Daniltsev, Denis Dimitrov, Anna Dmitrienko, Vladimir Dokholyan, Sergey Emelyanov, Dmitry Ermilov, Georgii Fedorov, Polina Gavrilova, Nikolai Gerasimenko, Aleksandr Gordeev, Andrey Inozemtsev, Andrei Ivaniuta, Alexander Ivanov, Mikhail Karaev, Anastasiia Kargapoltseva, Ivan Kirillov, Nikita Kiselev, Valeria Kobenko, Yury Kolabushin, Denis Koposov, Anatoly Korobov, Vladimir Korviakov, Kirill Kozlov, Denis Krzhivokolskiy, Konstantin Kuklev, Alexander Kunitsyn, Sergey Kuzin, Vladislav Lakhtionov, Alexey Letunovskiy, Maxim Litvinov, Alexander Lyulkov, Georgy Makarov, Kirill Malakhov, Egor Malykh, Mikhail Mamaev, Dmitrii Mikhailov, Polina Mikhailova, Ivan Mikheev, Elizaveta Muromtseva, Nikolai Nazarkin, Tatiana Nikulina, Lev Novitskiy, Stanislav Onuchin, Nikita Osterov, Denis Parkhomenko, Anatoliy Parpara, Vladimir Polovnikov, Konstantin Reznikov, Azat Saginbaev, Nikita Samsonov, Alexander Sentsov, Nikita Shaimov, Artem Sherstyuk, Andrey Shutkin, Egor Silvestrov, Bulat Suleimanov, Matvey Suprunov, Sergey Taranov, Irina Tolstykh, Tatiana Trofimuk, Ilya Trushkin, Aleksandra Tsybina, Olga Varlashina, Viacheslav Vasilev, Ilya Vasiliev, Eugeny Vilisov, Sergey Yakubson, Konstantin Zakharov

arXiv:2610.05608v1cs.CVcs.AIcs.LGcs.MM

TL;DR

Synchronized video-audio generation requires semantic, temporal, and emotional consistency, while closed commercial systems hinder reproducibility. Kandinsky 6.0 Video connects video and audio streams through bidirectional cross-attention and is competitive on several evaluated quality criteria, particularly speech.

  • Problem

    Synchronized video-audio generation must maintain semantic, temporal, and emotional consistency, and closed commercial systems have unavailable source code that hinders reproducibility and further research.

    Source: §1 · p. 4

  • Method

    Kandinsky 6.0 Video connects a pretrained video stream and an audio stream trained from scratch using blockwise bidirectional cross-attention, with separate modality pretraining followed by joint paired audio-video training.

    Source: §1 · p. 4, §10 · p. 62

  • Results

    On VABench, Kandinsky 6.0 Video Pro scores best among evaluated Kandinsky 6.0 Video Lite and LTX 2.5 models in speech quality, audio aesthetics, alignment, lip-sync, desynchronization, and visual realism.

    Source: §10 · p. 62

  • Takeaways & Limitations

    The reported comparisons support a criterion-specific position: Kandinsky 6.0 Video Pro is competitive with proprietary systems, while MiniMax H3 and Seedance 2.0 remain stronger on visual criteria.

    Source: §10 · p. 62

    Clips are limited to 5 seconds, base generation is SD resolution with Full HD supplied through super-resolution, and a gap remains to the strongest proprietary systems in visual and overall audio quality.

    Source: §10 · p. 62

Abstract

from arXiv · show

We present Kandinsky 6.0 Video, a family of foundation diffusion models for synchronized text-to-audio-video generation, comprising Kandinsky 6.0 Video Lite (3B parameters) and Kandinsky 6.0 Video Pro (29B parameters). Both models generate 5-second video clips with synchronized 44 kHz audio, including lip-sync, in text-to-audio-video (T2AV) and image-to-audio-video (I2AV) modes; a built-in super-resolution model raises the output resolution to Full-HD (1920$\times$1080). Building on the video generation capabilities of Kandinsky 5.0, Kandinsky 6.0 Video employs a dual-stream CrossDiT architecture that connects a pretrained video stream and a newly trained audio stream through bidirectional cross-attention for temporal and semantic alignment. Our continuous pretraining strategy first trains the audio stream from scratch on large-scale audio corpora and then trains both streams jointly on paired audio-video data while preserving unimodal fidelity; pretraining is followed by supervised fine-tuning, reinforcement-learning-based post-training, and distillation. In side-by-side human evaluation, Kandinsky 6.0 Video Pro clearly outperforms its predecessor, Kandinsky 5.0 Video Pro, and remains competitive with leading audio-video generation models, particularly in speech quality. To accelerate open research and deployment in multimedia generation, we release the code, model checkpoints, and diffusers integration under the MIT license.

1 Introduction

Kandinsky 6.0 Video addresses the difficulty of synchronized audio-video generation and the reproducibility limits of closed systems with two open model scales, a staged training pipeline, and competitive evaluation results. The paper also releases its weights, code, and diffusers integration under the MIT license.

1 Introduction

  • Synchronized audio-video generation requires semantic, temporal, and emotional consistency, while closed systems' unavailable source code hinders reproducibility and further research.The introduction notes that open-source solutions still lag closed systems in clip duration, resolution, and synchronization accuracy.Source: §1 · p. 4
  • Lite (3B parameters) and Pro (29B parameters) generate 5-second clips with synchronized 44 kHz audio, lip-sync, and text-to-audio-video or image-to-audio-video conditioning.Clips are generated at SD resolution and can be upscaled to Full-HD with a built-in super-resolution model.Source: §1 · p. 4
  • The models connect a pretrained video stream and a newly trained audio stream through bidirectional cross-attention, with audio pretraining followed by joint training on paired audio-video data.The post-training pipeline adds domain-specific supervised fine-tuning, reinforcement learning, and two-stage distillation to 10 function evaluations.Source: §1 · p. 4
  • Kandinsky 6.0 Video Pro clearly surpasses Kandinsky 5.0 Video Pro, beats LTX 2.5 on most VABench metrics, and remains competitive with proprietary systems, particularly in speech quality.The evaluation combines VABench with side-by-side human evaluation.Source: §1 · p. 4
  • The authors release all model weights, source code, and diffusers integration under the MIT license.The release is intended to support multimedia-generation research and deployment.Source: §1 · p. 4

2 Data Processing Pipeline

The data pipeline expands Kandinsky 5.0 infrastructure with modality-specific pretraining, fine-tuning, reinforcement-learning, and super-resolution datasets. It combines filtering and annotation procedures to prepare visual, audio, joint audio-video, and image-conditioned data for generation and alignment tasks.

2 Data Processing Pipeline

  • The expanded pipeline prepares image, video, audio, joint audio-video, image-conditioned, fine-tuning, reinforcement-learning, super-resolution, and Russian Cultural Code data, with infrastructure for thousands of concurrent processors and audio-visual embedding clustering.The infrastructure upgrades include table partitioning and load balancing.Source: §2 · p. 5, §2.1 · p. 5

2.2 Text-to-Image Dataset

  • The text-to-image corpus is a 50-million-image Kandinsky 5.0 subset re-captioned in Russian and English at four detail levels, with filtering for quality, diversity, minimal text, and visual complexity.The four caption levels are long, medium, short, and shortest.Source: §2.2 · p. 5, Dataset Properties · p. 5, Content Annotation · p. 5

2.3 Text-to-Video Dataset

  • The text-to-video dataset selects 20 million Kandinsky 5.0 scenes, excluding non-standard frame rates and scene transitions, and uses bilingual captions at four detail levels.Its filtering retains stable single-scene clips, diverse content, and controlled motion diversity.Source: §2.3 · p. 5, Dataset Properties · p. 6, Content Annotation · p. 6

2.4 Text-to-Audio Dataset; 2.5 Text-to-Audio-Video Dataset

  • Audio pretraining uses 40 million annotated tracks, while joint audio-video training uses 7 million segments filtered for audio quality, visual correspondence, and lip-sync suitability.For detected people, clips must meet an LSA threshold of 3/5; clips with audio-visual events must have Desync below 0.1, while face-free clips are retained.Source: §2.4 · p. 6, §2.5 · p. 6, Dataset Properties · p. 6

Captioning Pipeline; 2.6 Image-to-Audio-Video Dataset

  • Joint-training captions combine speech transcripts, audio descriptions, and visual descriptions, while the dedicated I2AV dataset pairs 116,000 starting frames with captions for expected audio-visual continuations.I2AV captions also incorporate camera-motion labels and are rewritten into four detail levels.Source: Captioning Pipeline · p. 7, §2.6 · p. 7

Dataset Properties

  • I2AV starting frames are screened for image quality and suitability as action beginnings or pauses, yielding a diverse dataset of semantically appropriate frames.The criteria include sharpness, artifacts, blur, visibility of the main object, and exclusion of empty or transitional frames.Source: Dataset Properties · p. 7

2.7 Supervised Fine-Tuning Datasets

  • SFT uses separate video-only and audio-visual datasets across 11 domains, with Stage 1 focused on visual quality and Stage 2 on synchronization and lip-sync.Stage 1 contains 4,663 scenes; Stage 2 contains 7,686 and adds speech and music content.Source: §2.7 · p. 8

2.8 Reinforcement Learning Dataset; 2.9 Super-Resolution Data; 2.10 Russian Cultural Code Dataset

  • The reinforcement-learning set contains 2,984 curated clips for online multi-objective training, and the 150,000-clip super-resolution set draws on three sources; the Russian Cultural Code collection adds 1.2 million images and 530,000 video scenes.The cultural dataset is used in T2V pretraining and injected through images during joint T2AV training.Source: §2.8 · pp. 8–9, §2.9 · p. 9, §2.10 · p. 9

3 Kandinsky 6.0 Video Architecture

Kandinsky 6.0 Video uses a dual-stream CrossDiT to extend Kandinsky 5.0’s video architecture to video and audio. Its separate latent-space upscaling components prepare higher-resolution latents and restore their detail for Full-HD output.

3.1 Overview

  • Both Lite (3B) and Pro (29B) use flow matching, with Hunyuan VAE for video and a 44 kHz-capable MMAudio autoencoder for audio.Both versions share the same architecture; their parameter allocations differ across video, audio, and cross-attention layers.Source: §3.1 · p. 9, §3.2 · p. 10

3.2 Core Model

  • The dual-stream CrossDiT connects asymmetric video and audio DiT backbones through blockwise bidirectional cross-attention.The streams inherit their architecture from Kandinsky 5.0 and use separate text encoders and timestep embeddings.Source: §3.2 · p. 10, Figure 2 · p. 12

3.3 Super-Resolution Model

  • The Full-HD super-resolution pipeline uses a Latent Upscaler to enlarge LQ latents and an SR-DiT to remove degradations and restore detail.The LU preserves temporal resolution, and its outputs remain degraded until SR-DiT processing.Source: §3.3 · p. 13, Figure 4 · p. 15

3.3.1 SR-DiT

  • SR-DiT processes noisy video latents with a first-frame HQ anchor and binary mask, predicts flow-matching velocity, and operates without text conditioning.The input channels are concatenated and processed by timestep-modulated DiT blocks.Source: §3.3.1 · p. 13, Figure 3 · p. 14

3.3.2 Latent Upscaler

  • The Latent Upscaler maps 64-channel LQ latents to higher-resolution 64-channel latents without changing temporal resolution.It has separate ×4 and ×2 branches; the ×2 branch uses a shorter front section to limit computation on its larger input grid.Source: Figure 4 · p. 15, §3.3.2 · pp. 15–16

4 Training Stages

Kandinsky 6.0 Video is trained through separate video and audio pretraining, joint audio-video training, and later post-training stages. In I2AV training, a reference frame is explicitly marked and assigned a temporal position; ablations find the mask and temporal RoPE more influential than Token Role Embeddings.

4.1 Overview

  • Training proceeds from separate video and audio pretraining to joint T2AV and mixed T2AV/I2AV training, followed by SFT, RL-based post-training, and distillation.The video stream starts from a pretrained Kandinsky 5.0 checkpoint, while the audio stream is trained from scratch before joint training.Source: §4.1 · p. 17, Figure 5 · p. 17, §4.2 · p. 17

4.2.1 Separate Training

  • Separate pretraining trains audio on audio captions and then audio-video captions, while video is pretrained in parallel from Kandinsky 5.0 checkpoints on audio-video captions.Lite uses 40k then 2k T2A steps and 20k T2V steps; Pro uses 30k then 4k T2A steps and 14k T2V steps.Source: §4.2.1 · pp. 17–18

4.2.2 Joint T2AV Training

  • Joint T2AV training runs for 57.5k steps for Lite and 49k steps for Pro, followed by additional mixed training with 25% I2AV samples.The mixed stage adds 4k steps for Lite and 5k for Pro.Source: §4.2.2 · p. 18

4.2.3 Image-to-Audio-Video (I2AV) Mode

  • In I2AV mode, the conditioning-frame latent is concatenated along the temporal axis, marked by a binary mask, and distinguished from generated-video tokens with learnable role embeddings.I2AV samples make up 25% of the final pretraining stage.Source: §4.2.3 · p. 18, §4.2.2 · p. 18, Figure 6 · p. 20
  • In I2AV ablations, the channel mask and temporal RoPE had the strongest effects, while Token Role Embeddings had a relatively weak effect and were partly redundant.An all-one mask almost blocked denoising; changing the reference frame’s RoPE position made it condition the corresponding video position.Source: §4.2.3 · pp. 18, 21

4.3 Supervised Fine-Tuning

Supervised fine-tuning proceeds in two stages across 11 domains, with domain-specific models averaged only after Stage 2. Stage 1 targets visual quality, while Stage 2 jointly improves synchronization and lip-sync.

Two-stage procedure

  • Each of the 11 domains is fine-tuned independently through both stages, and their models are uniformly weight-averaged only at the end of Stage 2.The same procedure is used for Kandinsky 6.0 Video Lite and Pro; the intermediate Stage 1 soup is used only for quality measurement.Source: §4.3 · p. 21

Stage 1 (visual quality)

  • Stage 1 freezes the audio, fusion, and audio-text modules and optimizes only video loss to improve visual quality.Audio inputs still pass through the frozen audio stream but do not contribute to the loss.Source: §4.3 · p. 21

Stage 2 (synchronization and lip-sync)

  • Stage 2 unfreezes all modules and combines video, audio, and facial-region losses before uniformly averaging the 11 domain checkpoints after 5,000 steps.The loss weights are 0.85 for video, 0.15 for audio, and 1.0 for face; the face loss focuses video loss on the facial latent region to encourage lip-sync.Source: §4.3 · p. 21

4.4 Reinforcement Learning

Reinforcement-learning post-training adapts OmniNFT to Kandinsky 6.0 Video using modality-aware rewards and regularization. It substantially improves speech intelligibility on held-out evaluations, while training uses text-to-audio-video prompts only.

RL approach

  • The adapted OmniNFT recipe combines modality-wise advantage routing, layer-wise gradient surgery, and region-wise loss reweighting.RL is performed online in pure T2AV mode, without image-conditioned generation.Source: §4.4 · p. 21

Reward functions and routing

  • Eight reward objectives are grouped into video, audio, and synchronization branches, with synchronization rewards contributing to both modality advantages.The reward phase centers and scales group scores, clamps advantages to [−5, 5], and converts them into per-token rewards.Source: §4.4 · pp. 22–23

RL setup

  • Pro uses 10 rollouts per prompt and Lite uses eight; both train LoRA adapters over a frozen SFT model, with prompts sampled in Russian or English and speech clips oversampled.RL rollouts use 512 × 768 resolution, below Full-HD inference resolution; speech clips are oversampled by 1.5×.Source: §4.4 · pp. 21–24

Optimization

  • The policy update compares current and old LoRA predictions against the clean rollout, weights per-token losses by rewards, and aggregates them with A2V-derived region attention.A KL term anchors the update to the frozen SFT model; the old LoRA is an exponential moving average of the current one.Source: Figure 8 · p. 23, §4.4 · p. 24

Regularization techniques

  • Gradient surgery scales video-branch gradients on the V2A key-value path, while region reweighting raises weights for video patches strongly attended by audio tokens.The gradient-scaling schedule is calibrated per model block, and region weights are capped at 1.5×.Source: §4.4 · p. 25

Results

  • Pro WER falls from 0.235 to 0.124 (47% relative reduction), while Lite WER falls from 0.179 to 0.127 (29%); Pro also drops from 0.251 to 0.180 on Harvard sentences.The held-out speech benchmark uses 64 prompts and three seeds each; the Harvard evaluation uses 100 prompts.Source: §4.4 · p. 25

4.5 Distillation

Distillation separates few-step trajectory compression from adversarial quality recovery. The paper selects on-policy π-Flow for trajectory distillation and introduces Sim-LADD to refine the compressed student using its own rollout states.

4.5 Distillation

  • The paper frames distillation as two complementary stages: trajectory compression into a few-step student, followed by recovery of perceptual fidelity with real data and adversarial training.Source: §4.5 · p. 25

4.5.1 Motivation and Method Selection

  • The paper reports that CFG-distillation and consistency methods can alter teacher motion dynamics and sample distributions, while remaining insufficiently flexible or performant for few-step video generation.Source: §4.5.1 · p. 25
  • Among tested methods, π-Flow showed the most reliable convergence and best teacher-trajectory preservation, training on states encountered during the student’s own rollout.Its on-policy formulation avoids the mode collapse and mode dropping observed with distribution-matching approaches and absorbs the guided teacher trajectory directly.Source: §4.5.1 · p. 26
  • Trajectory compression makes generated videos smoother and less detailed, with losses in high-frequency texture, local contrast, and fine spatial detail.The degradation increases with more aggressive compression even when semantic composition and global motion remain correct.Source: §4.5.1 · p. 26

4.5.2 Distillation Pipeline

  • π-Flow compresses the teacher into a 10-NFE student, while Sim-LADD recovers high-frequency detail and perceptual sharpness lost during compression.The two stages are complementary: π-Flow preserves semantics, motion, and diversity, and Sim-LADD refines perceptual fidelity using real data and adversarial learning.Source: §4.5.2 · pp. 26–27
  • Sim-LADD trains on the student’s backward-rollout states rather than forward-noised states, reducing the train–inference gap during adversarial refinement.The method backpropagates through the final five solver steps and stochastically re-noises real and generated latents before discriminator feature extraction.Source: §4.5.2 · p. 26, Figure 9 · p. 27

4.5.3 Training Details

  • The trajectory student starts from the pretrained teacher backbone and is trained with on-policy imitation regression against the guided teacher velocity field.Training moves through off-policy, mixed-policy, and on-policy regimes in a 1:1:2 ratio over 4,000 iterations.Source: §4.5.3 · p. 27
  • Sim-LADD uses a discriminator with heads attached to intermediate pretrained-backbone blocks, producing separate video and audio scores.Adversarial refinement uses a multi-scale latent-feature hinge loss, final-five-step gradient propagation, and stochastic re-noising before feature extraction.Source: §4.5.3 · pp. 27–28, Figure 10 · p. 28

4.6 Super-Resolution

This section describes a two-part super-resolution pipeline for restoring video latents at ×2 and ×4, and reports validation comparisons of latent upscaling against baselines using PSNR and warp error.

4.6 Super-Resolution

  • One SR-DiT is trained jointly at ×2 and ×4 with equal-probability scale sampling, while separate latent upscalers prepare its inputs.The SR-DiT and latent upscalers are optimized independently.Source: §4.6 · p. 28

4.6.1 Training Data and Degradations

  • Training pairs use accepted HQ crops and degraded counterparts, with Real-ESRGAN-style degradation [62] and CreativeVR-inspired temporal degradation.The Real-ESRGAN-style pipeline degrades image quality without changing frame sequence or explicitly warping geometry; the temporal pipeline introduces distortions to geometry, motion, and fine detail.Source: §4.6.1 · pp. 28–29

4.6.2 SR-DiT Training Objective

  • SR-DiT uses flow matching to restore HQ latents from degraded latents, with first-frame HQ anchor conditioning and decoded pixel losses targeting detail and grid artifacts.The objective combines reconstruction and anti-grid losses computed through the frozen K-VAE decoder.Source: §4.6.2 · pp. 29, 31, Figure 11 · p. 30

4.6.3 SR-DiT Training Stages

  • SR-DiT training progresses from dense attention to sparse NABLA attention, then π-Flow distillation [57] for inference in two network evaluations.The distillation student predicts velocity outputs for two trajectory segments, with the DX policy interpolating within each segment without additional network calls.Source: §4.6.3 · pp. 31–32

4.6.4 Latent Upscaler Training

  • The latent upscaler learns degraded-to-degraded latent upsampling, while SR-DiT handles restoration of clean HQ content.Training mixtures contain approximately 218k pairs for ×4 and 194k for ×2; temporal supervision matches frame-to-frame differences between predictions and targets.Source: §4.6.4 · pp. 32–33

4.6.5 Validation

  • At ×4, MUSIQ rose from 48.92 after Stage 1 to 50.38 and 50.41 after Stages 2 and 3; at ×2, it showed no consistent improvement.At ×4, warp error increased from 0.614 to 0.648 and 0.678, while the Laplacian artifact score changed from 0.437 to 0.534 and then 0.423. Metrics are complementary, and final checkpoint selection uses side-by-side human evaluation; scores across scales are not directly comparable.Source: Table 4 · p. 34, §4.6.5 · p. 34
  • ×4 LU has higher mean PSNR than pixel-space bilinear upsampling in all four groups; at ×2 it wins two groups, while pixel-space bilinear wins two.Source: §4.6.5 · p. 35
  • On 94 clips, LU mean warp error is 0.684 at ×2 and 0.626 at ×4, versus 0.638 and 0.587 for pixel-space bilinear upsampling.Warp error depends on spatial detail and optical-flow accuracy, and the degraded targets may contain temporal artifacts from the degradation pipeline.Source: §4.6.5 · p. 35
  • Latent-space bilinear interpolation followed by decoding yields 21.44 dB at ×2 and 19.76 dB at ×4, indicating poor target reconstruction.Source: §4.6.5 · p. 35
  • On 24 real clips, LU increases mean warp error over the approximately 0.591 round-trip baseline by 0.026 at ×2 and 0.049 at ×4.Both routes were evaluated on a common output canvas; the ×2 route included pixel-space pre-upsampling. These results do not uniquely attribute temporal error to the VAE or LU; textile had the highest absolute warp error for both routes.Source: §4.6.5 · p. 35

5 Inference

The inference pipeline offers SD, HD, and Full HD output, using tiled latent-space super-resolution for higher-resolution modes. Its final distilled SR model uses π-Flow DX sampling with two network evaluations per tile.

5 Inference

  • Full HD output uses a ×2.25 super-resolution route, then resizes to the target dimensions.The route combines pixel-space ×1.125 pre-upscaling with the ×2 route.Source: §5 · p. 36, Figure 12 · p. 36
  • For the ×2.25 route, pixel-space pre-upscaling aligns dimensions to K-VAE's 16-pixel spatial compression factor before applying ×2 super-resolution.An example maps 864×480 to 976×544 before the ×2 route.Source: §5 · p. 36
  • The clip is encoded once, split into overlapping spatial latent tiles, upscaled to the SR-DiT working resolution, decoded, and blended.The pipeline uses at least 25% tile overlap and normalized Hann-window blending to reduce boundary seams.Source: §5 · pp. 36–37, Figure 13 · p. 37
  • The final distilled model replaces the pre-distillation denoising loop with π-Flow DX policy sampling at NFE = 2 per tile.The policy uses network-free Euler substeps between two network evaluations per tile.Source: §5 · pp. 37–38, Figure 13 · p. 37
  • Table 6 reports processing times for 5-second clips on one H100 GPU using the distilled SR-DiT and scale-specific Latent Upscalers.Reported timings include K-VAE encoding and exclude the final resize; tiles are processed sequentially with batch size one.Source: Table 6 · p. 38, §5 · p. 38

6 Optimizations and Consumer GPU Deployment

Block offloading reduces peak memory so Lite and Pro can run on 16, 24, and 32 GB cards without changing the mathematical model. The section also reports output similarity under SR memory optimizations and describes deployment presets and inference accelerations.

6.1 Block Offloading

  • Block offloading lets Lite and Pro run on 32, 24, and 16 GB cards by keeping only two transformer blocks on the GPU at a time.It reduces Pro peak memory by about 3.4× at SD and 1.8× at HD and Full HD, while the mathematical model remains unchanged.Source: §6 · p. 38

6.2 VRAM Budget Presets

  • At the VRAM budget presets, peak allocated memory is the same for Lite and Pro because only two DiT blocks remain resident.The 16 GB preset also splits SR decoding spatially and quantizes the text encoder; Pro requires about 58 GiB process RSS.Source: §6.2 · p. 39, Table 9 · p. 40

6.3 Inference Acceleration

  • Inference acceleration combines regional DiT compilation, MagCache step skipping, GPU-specific attention backends, and regional SR VAE decoder compilation.On Pro SD, MagCache evaluates 49 of 100 forwards and roughly halves the base stage.Source: §6.3 · p. 40

6.4 Performance

  • On the RTX 5090, SD and Full HD base-stage times agree within 0.5%, while the SR stage takes around 64 seconds at HD and 96 seconds at Full HD.The base modes share geometry; SR time depends on resolution, not Lite versus Pro.Source: §6.4 · p. 41

6.5 Quality Under Optimization

  • Approximately 12% of pixels differ by one brightness level under SR memory settings, with PSNR around 57 dB and no visible structure or seams at segment boundaries.The tested settings include segment length, spatial strips, and offloading.Source: §6.5 · p. 41

7 Results

Kandinsky 6.0 Video Pro leads on several VABench dimensions and clearly outperforms Kandinsky 5.0 Video Pro, while comparisons with other systems show criterion-specific strengths. The results also assess reinforcement learning and distillation through side-by-side human comparisons.

7.1 Benchmarking

  • Pro leads on speech quality, audio aesthetics, text-video and audio-video alignment, lip-sync, desynchronization, and visual realism; Lite leads on text-audio alignment, audio realism, and audio and visual quality.LTX 2.5 leads on VLM-judged alignment and expressiveness and on word error rate; the reported VABench WER uses a different ASR setup from the separate RL speech benchmark.Source: §7.1 · p. 41

7.2.3 Kandinsky 6.0 Video Pro vs Kandinsky 5.0 Video Pro

  • Kandinsky 6.0 Video Pro consistently leads Kandinsky 5.0 Video Pro across visual and semantic criteria in T2V and I2V, including reference-image correspondence in I2V.The side-by-side protocol anonymizes videos, randomizes their positions, and aggregates annotator votes; questions cover visual criteria first, then audio-visual criteria when applicable.Source: §7.2.1 · pp. 41, 43, §7.2.3 · p. 44, Figure 15 · p. 45

7.2.4 Kandinsky 6.0 Video Pro vs Kling 2.6

  • Kling 2.6 leads on visual aesthetics and overall audio quality, while Kandinsky 6.0 Video Pro leads on sound-prompt following and I2AV camera motion.Speech quality and task solving are near parity; synchronization and overall audio quality moderately favor Kling 2.6.Source: §7.2.4 · pp. 44, 46

7.2.5 Kandinsky 6.0 Video Pro vs LTX 2.5

  • Kandinsky 6.0 Video Pro leads LTX 2.5 on visual and speech-related dimensions, while LTX 2.5 has a small, non-significant edge in general audio fidelity.Camera motion is near parity, and most prompts did not request camera movement.Source: §7.2.5 · p. 46

7.2.6 Kandinsky 6.0 Video Pro vs Veo 3.1 Fast

  • Kandinsky 6.0 Video Pro and Veo 3.1 Fast trade strengths: Kandinsky has cleaner visuals and better input-image preservation, while Veo better follows prompts and delivers stronger audio.The comparison reports no overall dominant model; many criterion-level differences are statistically significant in the respective modes.Source: §7.2.6 · p. 46, Figure 18 · p. 49

7.2.7 Kandinsky 6.0 Video Pro vs MiniMax H3

  • MiniMax H3 is stronger overall, leading on most criteria in both modes, while Kandinsky 6.0 Video Pro remains competitive on synchronization and speech-related criteria.H3's significant leads include visual quality, artifacts, motion realism, visual prompt following, overall audio quality, sound-prompt following, and task solving.Source: §7.2.7 · pp. 46, 50, Figure 16 · p. 47

7.2.8 Kandinsky 6.0 Video Pro vs Seedance 2.0

  • Seedance 2.0 leads on visual criteria, while Kandinsky 6.0 Video Pro leads on speech quality and is competitive on synchronization.The speech advantage also appears in I2AV; synchronization and sound-prompt following are close to parity.Source: §7.2.8 · p. 50, Figure 20 · p. 52

7.2.9 Ablation: Reinforcement Learning vs SFT; 7.2.10 Ablation: Distillation vs Full Model

  • RL improves all three settings; the distilled model is preferred or tied on most criteria, averaging 51% vs 49% without significant differences.RL gains span speech-focused T2AV, general-domain T2AV, and general-domain I2AV; distillation preserves full-model quality while reducing inference cost.Source: §7.2.9 · p. 50, §7.2.10 · p. 54, Figure 22 · p. 54

8 Examples

The examples illustrate synchronized audio-visual generations and show how super-resolution adds detail to selected video crops. The authors note that printed figures cannot fully convey playback or temporal alignment.

8.1 Audio-Visual Examples

  • Audio-visual examples pair sampled frames with waveforms to make timing and scene-consistent sounds visible, complementing benchmark and human-evaluation results.The examples span animated scenes, speaking characters, and impact-synchronized sound.Source: §8 · pp. 54–55, §8.1 · p. 55
  • Figures 23–29 illustrate generated scenes with speech, ambience, and action-linked audio, including an astronaut, a detective dog, and a laboratory scene.Other examples include a musician, hammering, a child speaking, and a pirate shouting.Source: §8.1 · p. 55, Figure 23 · p. 55, Figure 24 · p. 55

8.2 Super-Resolution Examples

  • The full super-resolution pipeline sharpens boundaries and textures in selected crops, with ×4 outputs showing more pronounced detail than ×2 outputs.The examples include an animated painting, a tennis scene, and an interview; latent-upscaler outputs remain visually close to enlarged inputs.Source: §8.2 · p. 55, Figure 30 · p. 58, Figure 31 · p. 59

9 Related Work

Related work spans image and video generation, joint audio-video models, post-training and few-step sampling, and video super-resolution. Kandinsky 6.0 Video builds on the Kandinsky video line and uses a dual-stream approach for joint audio-video generation.

9.1 Image and Video Generation

  • Kandinsky 6.0 Video builds on the CrossDiT-based Kandinsky 5.0 family and prior work on video transformers, efficient attention, autoencoders, temporal control, and culture-specific generation.The Russian Cultural Code data used here is supported by culture-specific datasets and benchmarks.Source: §9.1 · p. 61

9.2 Joint Audio-Video Generation

  • Joint audio-video systems use varied designs, while Kandinsky 6.0 Video initializes its video stream from a pretrained model and trains its audio stream from scratch.Prior approaches include flow-based generation, hierarchical synchronization, tri-modal transformers, and dual-stream cross-modal fusion.Source: §9.2 · p. 61

9.3 Post-Training and Few-Step Sampling

  • Prior post-training work includes model soups, reinforcement learning for preference alignment, and several distillation approaches to reduce sampling steps.The cited approaches include progressive distillation, consistency models, distribution matching, adversarial distillation, and policy-based imitation distillation.Source: §9.3 · p. 61

9.4 Video Super-Resolution

  • Kandinsky's video super-resolution model combines a deterministic latent upscaler with a text-free diffusion transformer distilled to two network evaluations.Prior work addresses synthetic-degradation training, diffusion priors, temporal artifacts, and video-to-video restoration.Source: §9.4 · pp. 61–62

10 Conclusion

Kandinsky 6.0 Video combines pretrained video and newly trained audio streams for synchronized text-to-audio-video and image-to-audio-video generation. The authors report strong results on several criteria, alongside limits on clip length and native resolution and risks associated with generated content.

10 Conclusion

  • Lite (3B parameters) and Pro (29B) connect video and audio streams through blockwise bidirectional cross-attention and use continuous pretraining, fine-tuning, reinforcement learning, and distillation.A separate latent-space super-resolution model raises resolution to Full HD; block offloading and memory presets support use on consumer GPUs with 16–32 GB memory.Source: §10 · p. 62
  • Human evaluation shows Pro clearly outperforming Kandinsky 5.0 Video Pro; it is competitive with proprietary systems, and reinforcement learning reduces Pro speech WER by 47%.On VABench, Pro leads the evaluated models on speech quality, audio aesthetics, text-video and audio-video alignment, lip-sync, desynchronization, and visual realism.Source: §10 · p. 62
  • The models generate clips of up to 5 seconds, start at SD resolution before super-resolution, and retain a gap to the strongest proprietary systems in visual and overall audio quality.The authors identify longer clips, higher native resolution, and reference-based conditioning as future directions.Source: §10 · p. 62

10 Conclusion; Ethical Considerations

  • The authors release model weights, source code, and diffusers integration under the MIT license to support further multimodal-generation research.They also warn that realistic synchronized speech and lip movements can be misused for deception, impersonation, disinformation, fraud, and harassment.Source: §10 · p. 62, Ethical Considerations · p. 62

Ethical Considerations

  • Training-data biases may yield uneven representation, and outputs may contain factual, physical, or cultural inaccuracies rather than depict real events.The authors encourage research into these limitations and reporting of problematic behavior.Source: Ethical Considerations · p. 63
Loading 2610.05608v1…