Source-linked AI summary
Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion
Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, Denis Dimitrov
TL;DR
Text-to-image generation has advanced through diffusion models, which can operate at pixel or latent level. Kandinsky combines an image prior with latent diffusion and achieves a reported FID of 8.03 on COCO-30K, while offering free-to-use interfaces. The authors note that semantic coherence and image-quality measures still need improvement.
Problem
Method
Kandinsky maps text embeddings to CLIP image embeddings with an image prior and then generates images through latent diffusion.
Source: §4 · pp. 3–4Results
8.03 FID on the COCO-30K validation set at 256×256, placing Kandinsky in close competition with state-of-the-art models and among top open-source systems.
Source: §6 · p. 5Takeaways & Limitations
The authors present Kandinsky as a system for image generation and processing with free-to-use web and Telegram interfaces.
Source: §7 · p. 6The authors identify semantic coherence with input text and absolute FID and human-evaluation image-quality values as areas needing improvement.
Source: §8 · p. 6
Abstract
from arXiv · showhide
Text-to-image generation is a significant domain in modern computer vision and has achieved substantial improvements through the evolution of generative architectures. Among these, there are diffusion-based models that have demonstrated essential quality enhancements. These models are generally split into two categories: pixel-level and latent-level approaches. We present Kandinsky1, a novel exploration of latent diffusion architecture, combining the principles of the image prior models with latent diffusion techniques. The image prior model is trained separately to map text embeddings to image embeddings of CLIP. Another distinct feature of the proposed model is the modified MoVQ implementation, which serves as the image autoencoder component. Overall, the designed model contains 3.3B parameters. We also deployed a user-friendly demo system that supports diverse generative modes such as text-to-image generation, image fusion, text and image fusion, image variations generation, and text-guided inpainting/outpainting. Additionally, we released the source code and checkpoints for the Kandinsky models. Experimental evaluations demonstrate a FID score of 8.03 on the COCO-30K dataset, marking our model as the top open-source performer in terms of measurable image generation quality.
1 Introduction
Kandinsky combines image-prior mapping with latent diffusion in a text-to-image architecture. The paper reports top open-source image-generation results and presents software and an interactive image editor.
1 Introduction
- The introduction frames the work amid rapid improvements in text-to-image models, including photorealistic quality, near-real-time inference, and expanding applications.It notes the growth of both easy-to-use web platforms and sophisticated AI graphics editors.Source: §1 · p. 1
- Kandinsky combines image prior and latent diffusion in a text-to-image architecture.The paper describes this as the first architecture designed with that combination.Source: §1 · p. 1
- The model achieves state-of-the-art FID results among open-source systems, comparable to Stable Diffusion, IF, and DALL-E 2.The paper reports comparisons in terms of FID.Source: §1 · p. 1
- The authors provide a software implementation and release pre-trained models under Apache 2.0, permitting commercial and non-commercial use.The paper describes this release as unique among top-performing methods.Source: §1 · p. 1
- The web image editor supports interactive English- and Russian-language prompting and inpainting and outpainting.The paper also describes the editor as an image-generation application.Source: §1 · p. 1
2 Related Work
Related work traces text-to-image generation from autoregressive models to diffusion approaches. It distinguishes pixel- and latent-level diffusion and describes diffusion models’ broader applications.
2 Related Work
- Early autoregressive text-to-image models often suffered from content-level artifacts, while diffusion models subsequently became central to the field.Examples include DALL-E and CogView among autoregressive models, and DALL-E 2, Imagen, and Stable Diffusion among diffusion models.Source: §2 · p. 2
- Diffusion approaches are used in applications including 3D object synthesis, video generation, and controllable image editing.The passage also mentions DreamBooth and DreamPose as open-source generative-model applications.Source: §2 · p. 2
- Text-to-image diffusion models use different text encoders and may generate low-resolution images in pixel space before diffusion-based upsampling.The passage names T5 in Imagen and CLIP in GLIDE and DALL-E 2.Source: §2 · p. 2
3 Demo System
The demo system provides a Telegram bot and a web-based image editor with multiple image-generation and editing workflows. The web editor supports text prompts, inpainting, and outpainting.
3 Demo System
- The system implements two main inference resources: a Telegram bot and the FusionBrain website.The website is a web-based image editor with image loading and saving, erasing tools, zooming, and style selection.Source: §3 · pp. 2–3
- The web editor supports text-to-image generation from Russian or English prompts, with five selectable aspect ratios.The listed ratios are 9:16, 2:3, 1:1, 16:9, and 3:2.Source: §3 · p. 3
- Inpainting fills an erased image region with or without text guidance, while outpainting extends an image using a sliding-window mask.When the outpainting window intersects the image, its empty portion is generated, optionally with text guidance.Source: §3 · p. 3
- The Telegram bot offers text-to-image generation, image fusion, image-and-text fusion, and image variations.Image fusion combines a main image with a guiding image; variations are several images similar to the input.Source: §3 · p. 3
4 Kandinsky Architecture
Kandinsky combines text encoding, an image prior that maps text and image embeddings, and latent diffusion. Its design also uses a custom MoVQGAN autoencoder and multiple text-conditioning signals.
4 Kandinsky Architecture
- Kandinsky combines CLIP-text with image-prior mapping and XLMR text embeddings, while its latent diffusion conditions on CLIP-image, CLIP-text, and XLMR-CLIP embeddings.CLIP-image and XLMR-CLIP embeddings are merged as input to latent diffusion, and all listed embeddings are added to the time embedding.Source: §4 · pp. 3–4
- Kandinsky's three stages are text encoding, embedding mapping through an image prior, and latent diffusion.The focus on latent diffusion was motivated by its training efficiency relative to pixel-level diffusion.Source: §4 · p. 3
- The image prior uses a transformer encoder trained with diffusion on CLIP-ViT-L14 text and image embeddings, with visual embeddings normalized using full-dataset statistics.The normalization leads to faster convergence, and inverse normalization restores the original CLIP-image embedding space at inference.Source: §4 · p. 4
- The latent diffusion component uses a UNet and a custom pre-trained autoencoder, and retains autoencoder quantization during inference.The authors report that retaining quantization increases generated-image diversity and quality.Source: §4 · p. 4
- Sber-MoVQGAN is the authors' modified MoVQGAN implementation, developed because image decoding was their main image-quality bottleneck.The autoencoder was trained on LAION HighRes; the comparison with competitors is presented in Table 4.Source: §4 · pp. 4–5
5 Experiments
The experiments evaluate Kandinsky's image-generation quality with automatic metrics and blind human judgments. They compare image-prior configurations and test the effect of latent quantization.
5 Experiments
- The experiments use FID-CLIP curves on COCO-30K to select a guidance scale and compare Kandinsky with competitors.The prior configurations include no prior, a one-layer linear prior, an 18-block ResNet prior, and a transformer diffusion prior.Source: §5 · p. 5
- The prior ablation compares direct text-embedding use, linear mapping, an 18-residual-block ResNet, and a transformer diffusion prior.These configurations were explored to assess their effect on performance.Source: §5 · p. 5
- The study tests MoVQ latent quantization with quantization enabled and disabled to assess its influence on image-generation quality.The paper also reports that the model retains quantization during inference because it increases diversity and quality.Source: §5 · p. 5
- The evaluation includes FID scores for IF, described as the closest open-source competitor to Kandinsky.The authors computed IF's FID for comparison.Source: §5 · p. 5
- A blind human evaluation on DrawBench supplements automatic metrics because those metrics do not capture nuances of user experience.The authors use the human assessment alongside FID-CLIP curves to evaluate generated-image quality from a human-perception perspective.Source: §5 · p. 5
6 Results
Kandinsky's reported COCO-30K result places it among the top open-source systems and close to state-of-the-art models. The experiments also distinguish which prior performs best on FID versus CLIP and human evaluation.
6 Results
- 8.03 FID on COCO-30K validation at 256×256 came from Linear Prior, placing Kandinsky among top open-source performers and close to state-of-the-art models.The authors identify Linear Prior as the configuration with the best FID.Source: §6 · pp. 5–6
- The diffusion prior obtains the best CLIP score and human-evaluation score.The paper reports these as distinct from the best FID result, which uses Linear Prior.Source: §6 · p. 6
- For image quality, the reported comparison gives FID values of 9.86 versus 9.87; the best CLIP score and human-evaluation score are obtained by the diffusion prior.Source: §6 · p. 6
- A linear mapping trained on 500 cat images, termed the cat prior, displayed high proficiency on the prompt “astronaut riding a horse.”The authors present this experiment as further scrutiny of the hypothesis that visual and textual embedding spaces may have a linear relationship.Source: §6 · p. 6
7 Conclusion
Kandinsky is a latent-diffusion system for image generation and processing that the authors report achieves state-of-the-art results among open-source systems. They also describe an image-prior ablation study, free-to-use interfaces, released models and code, and directions for future research.
7 Conclusion
- The system supports image generation and processing tasks through a novel latent diffusion model.Source: §7 · p. 6
- Kandinsky yielded state-of-the-art results among open-source systems.The authors also report an extensive ablation study of image-prior design choices.Source: §7 · p. 6
- Free-to-use interfaces are available as a web application and Telegram bot.Source: §7 · p. 6
- Pre-trained models are available on Hugging Face, and the source code has a permissive license that allows commercial applications.Source: §7 · p. 6
- Future work includes exploring newer image encoders, more efficient UNets, higher resolutions, and improved prompt understanding.The authors also list local text-guided editing, attention reweighting, physics-based control, and moderation against abusive outputs.Source: §7 · p. 6
8 Limitations
The authors identify semantic coherence and image quality as areas for further improvement. They specifically call for better absolute FID values and image quality in human evaluations.
9 Ethical Considerations
The authors report cleansing training data and detecting abusive textual prompts as steps to reduce harmful outputs. They caution that carefully engineered prompts may still produce undesirable content and recommend additional safeguards for some applications.
9 Ethical Considerations
- The authors sought to reduce harmful outputs by cleansing marked training samples and detecting abusive textual prompts.Source: §9 · p. 7
- Carefully engineered prompts are not guaranteed to avoid undesirable content, despite obvious queries almost never producing abusive content in the authors’ tests.Source: §9 · p. 7
- Depending on the application, the authors recommend classifiers to filter undesirable content or tailored image and representation transformations.Source: §9 · p. 7