Source-linked AI summary
KVAE: Family of Tokenizers for Multimodal Generative Models
Andrey Shutkin, Denis Parkhomenko, Ivan Kirillov, Kirill Chernyshev, Kirill Malakhov, Ilia Vasiliev, Ilia Trushkin, Valeriya Kobenko, David Chikovani, Alexander Ivanov, Azat Saginbaev, Egor Silvestrov, Ivan Mikheev, Konstantin Zakharov
TL;DR
Latent diffusion depends on tokenizer representations, whose reconstruction quality may not predict downstream generation quality. KVAE develops tokenizers for audio, image, and video, and reports competitive reconstruction and generation results across modalities, including a 5.3-point semantic-quality advantage for one image-generation comparison.
Problem
Latent diffusion depends on tokenizers whose reconstruction quality does not necessarily predict downstream generation quality.
Method
KVAE develops tokenizers for audio, image, and video and evaluates candidates with diffusability diagnostics and downstream generative models.
Results
KVAE matches or surpasses open-source alternatives across reconstruction and generation evaluations, including a 5.3 percentage-point semantic-quality advantage over the FLUX.2 VAE stack.
Takeaways & Limitations
The released KVAE tokenizers and training details provide open-source components and guidance for text-conditioned generation across modalities.
Takeaways & Limitations
Audio FAD scores are insensitive to frequencies above 16 kHz, so they do not evaluate full-band fidelity.
Abstract
from arXiv · showhide
Latent diffusion modeling (LDM), a prominent paradigm, utilizes tokenizers to map input signal to compressed representation. This dependency positions tokenizer as an integral part of generation process itself, since it affects learning speed, quality of synthesized samples and lay foundation for later applications. This report presents series of KVAE tokenizers for audio, image and video, all designed for subsequent text-conditioned generation: KVAE-Audio, a continuous full-band 48 kHz tokenizer with a 50 Hz latent of 64 channels; KVAE-3D -- two causal video tokenizers for 4x16x16 and 4x8x8 compression; KVAE-2D, an image model, compressing input by factor of 8 with 32 channels. We demonstrate that reconstruction (PSNR, LPIPS, PESQ, etc.) and generation results on objective (Frechet Distance, CLIP score, CLAP score, etc.) and subjective (side-by-side evaluation) metrics matches or surpasses frontier opensource tokenizers, such as VAEs from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio. Considering difficulty of development, we share with community training details, model selection method and ablation on design choices. The code is publicly available at https://github.com/kandinskylab/kvae and https://github.com/kandinskylab/kvae-audio.
1 Introduction
Latent diffusion models rely on visual tokenizers to compress signals into latent spaces, making tokenizer design central to generation quality and efficiency. KVAE presents multimodal tokenizers targeting diffusability and high compression across visual and audio domains.
- Motivation: Latent diffusion models generate in compressed latent spaces formed by tokenizer encoder–decoder pairs.The tokenizer encodes the input and decodes generated latents back into the original signal domain.
- Motivation: Diffusability describes whether a tokenizer forms a latent space suitable for diffusion modeling.Tokenizer quality is therefore not determined solely by reconstruction fidelity.
- Motivation: High compression reduces the number of tokens processed by attention, potentially speeding training and inference.The motivation links this efficiency to reduced computational cost and carbon emissions.
- Contribution: KVAE introduces tokenizers for multiple modalities while comparing them with open-source alternatives on reconstruction and generation.The report also discusses design problems and development choices across modalities.
2 Related Works
Prior work spans adversarial, autoregressive, and diffusion-based generation, with latent diffusion making tokenizer properties a central design concern. Recent studies motivate evaluating diffusability through inexpensive diagnostics because reconstruction quality does not reliably predict generation quality.
- Generation paradigms: Generative video research progressed from GANs to autoregressive Transformers and diffusion models conditioned on text.These approaches expanded from pixel-level or short-video generation toward larger-scale latent-space systems.
- Tokenization: Latent diffusion introduced a dependency on tokenizer compression and representation quality for downstream generation.This dependency has motivated modifications involving compression ratio, foundation-model features, and decoder design.
- Tokenization: Reconstruction metrics do not reliably predict downstream generation quality, creating a reconstruction–generation dilemma.Under fixed downstream compute, higher per-token feature dimension can improve reconstruction FID while worsening generative FID.
- Diffusability: Diffusability concerns latent amenability to diffusion modeling rather than decoder reconstruction accuracy.Frequency and correlation statistics are candidate diagnostics, not the definition of diffusability.
- Diffusability: KVAE evaluates spatial correlation decay as an inexpensive frozen-tokenizer screening signal for downstream generation quality.The statistic is tested as an initial candidate criterion for balancing spectral bias and information capacity.
3 Method
KVAE defines causal video tokenizers as encoder–decoder systems with configurable temporal and spatial compression, then trains them through staged reconstruction, adversarial, spectral, and decoder objectives. The models use attention-free Conv3D architectures and sequence-length scaling for long videos.
- Architecture: A visual tokenizer encodes RGB video into a compressed latent and decodes it back to the original video dimensions.Temporal and spatial latent sizes are determined by compression factors and channel count.
- Architecture: KVAE-4×8×8 uses 16 channels, while KVAE-4×16×16 uses 64 channels.The notation records temporal compression, spatial compression, and hidden-channel count.
- Architecture: The video models use attention-free Conv3D blocks with caching to support efficient training and arbitrarily long inference sequences.Spatial RMSNorm permits segment-size adjustment during inference.
- Training: KVAE trains on progressively longer sequences, beginning at 65 frames and capping training sequences at 129 frames.The reported schedule improved convergence and generalized to longer sequences.
- Training: Training proceeds through reconstruction and KL losses, frame-based GAN refinement, EQ-loss with latent transformations, and decoder finetuning.The first-stage objective combines mean absolute error, LPIPS, and KL divergence.
- Data: The tokenizer dataset contains 10 million images and 2 million videos selected from the generative model’s pretraining data.Preprocessing is limited to flips, rotations, resizing, and cropping across resolutions from 256 to 512.
4 Results
Across video, image, and reconstruction evaluations, KVAE generally matches or exceeds open-source baselines, while exposing a gap between objective reconstruction or generation metrics and human preference. KVAE-4×16×16 also forms a latent space associated with faster convergence and better video generations than HunyuanVideo-1.5 VAE.
- Video reconstruction: More than 1 dB average PSNR gain is reported for KVAE video tokenizers on MCL-JCV reconstruction.The comparison is against leading open-source methods including HunyuanVideo and Wan.
- Image generation: Higher compression factors produce faster text-to-image convergence, and KVAE-4×16×16 with 64 channels surpasses HunyuanVideo-1.5’s 32-channel FID result.The finding leaves open the relationship between compression ratio and generative performance.
- Image generation: KVAE-4×8×8 leads side-by-side evaluation despite contradictory objective metrics.This result illustrates that objective scores and human preference can disagree.
- Image generation: KVAE-4×16×16 is preferred over HunyuanVideo-1.5 across prompt following, visual quality, and semantic quality, but trails Wan-2.2 in visual appearance.Its side-by-side preference pattern is consistent with its objective metrics.
- Video generation: KVAE-4×16×16 forms a better video-generation latent space than HunyuanVideo-1.5 VAE, leading to faster convergence and better generations in the Kandinsky-5 pipeline.The evaluation uses MovieGen Benchmark prompts and multimodal generation metrics.
- Image reconstruction: KVAE-2D-2.0 surpasses FLUX.1-dev and FLUX.2-dev on PSNR, SSIM, and NED for text-rich document reconstruction.The benchmark contains 3042 images from OmniDoc-TokenBench.
- Image generation: 5.3 percentage points is the largest point-estimate advantage for the KVAE-2D-2.0 downstream stack over the FLUX.2 VAE stack in semantic quality.The comparison uses two 2B T2I DiT checkpoints after 200K optimization steps.
5 Analysis
The analysis develops inexpensive tokenizer-screening diagnostics, especially CDS, and evaluates design choices using reconstruction and downstream generation behavior. It also shows that practical architectural substitutions can preserve quality, while channel width and decoder capacity create reconstruction–generation trade-offs.
- Diffusability screening: CDS was retained as the primary spatial statistic for screening frozen tokenizers against downstream DiT visual quality.The study compares spatial self-similarity, semantic, spectral, and PCA-based diagnostics before focusing on CDS for model selection.
- Diffusability screening: r = 0.906 Pearson correlation links CDS with Bradley–Terry visual-quality scores across 14 image-tokenizer configurations.The association supports candidate screening but does not establish out-of-sample prediction.
- Diffusability screening: CDS increased from 0.0429 at initialization to 0.0518 after 400k steps along one joint VAE–DiT trajectory.The single trajectory showed steeper correlograms, but does not validate checkpoint-level subjective quality or establish a general training law.
- Normalization: RMSNorm preserves the learning dynamics and quality of GroupNorm while avoiding temporal-statistics causality violations and all-to-all synchronization.The comparison concerns KVAE-4x8x8 and motivates the more practical normalization choice.
- Decoder width: Equalizing encoder and decoder width improves reconstruction metrics but loses pace in generation, as reflected by gFID and CLIP-score curves.This ablation motivates the wider decoder used in KVAE-4x16x16 despite its decoding-complexity cost.
- Channel width: Reducing KVAE-4x16x16 from 64 to 32 channels worsens validation reconstruction by up to 1dB PSNR and 0.01 LPIPS, while generation remains behind the baseline.The 32-channel model’s visual quality consistently lags, although final CLIP-scores can reach the same values.
6 Audio tokenization
KVAE-Audio is a continuous full-band waveform tokenizer designed to provide a diffusible latent for text-to-audio generation. It combines 48 kHz operation, a 50 Hz 64-channel latent, foundation-model alignment, and evaluation through both reconstruction and downstream generation.
- Design requirements: KVAE-Audio models waveforms end-to-end at 48 kHz, avoiding the bandwidth and phase limitations of mel-spectrogram tokenizers with separate vocoders.The design targets media production and joint audio-video generation, where full-band audio is required.
- Architecture: KVAE-Audio uses a 50 Hz latent with 64 channels, corresponding to temporal compression factor ft = 960 and one latent frame per 20 ms.Its convolutional backbone compresses time through strided convolutions and widens the representation in channels.
- Latent selection: Foundation-model alignment improves latent diffusability and prompt following without a noticeable reconstruction-quality cost.Candidate selection uses a cheap correlation-decay screening signal, but final selection follows training a generative model on each frozen tokenizer.
- Reconstruction: KVAE-Audio leads spectral reconstruction distances on general audio and music and remains within noise of the best model on speech, using 64 channels and 166.9M parameters.It is compared with released MMAudio, MovieGen Audio, and SAME-L tokenizers under an identical evaluation protocol.
- Generation: KVAE-Audio consistently outperforms compared open tokenizers on AudioCaps, while results on music and speech vary by metric and domain.On music, MMAudio retains advantages on FAD and CLAP; on speech, KVAE-Audio leads on FAD (PANNs), CE, and intelligibility.
- Design trade-offs: The 64-channel setting balances reconstruction and generation because fewer channels visibly degrade reconstruction, whereas wider latents slow diffusion convergence and worsen fixed-budget generation.The crossing point depends on the tokenizer–generator pair and is not an intrinsic property of audio latents.
7 Conclusion
The report presents KVAE tokenizers for text-conditioned generation across visual and audio domains, surpassing leading alternatives on corresponding benchmarks.
- KVAE tokenizers surpass leading alternatives on corresponding visual and audio benchmarks.