Source-linked AI summary
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, Yaniv Taigman
TL;DR
Existing text-to-image methods improve fidelity and text relevance but lack broader controllability and other capabilities needed for applicability and quality. The paper introduces scene conditioning, perceptually targeted tokenization, and transformer classifier-free guidance, achieving state-of-the-art FID and human-evaluation results while generating 512x512 images. The resulting system supports scene editing, anchored text editing, out-of-distribution generation, and story illustration.
Problem
Existing text-to-image methods improve fidelity and text relevance but lack several pivotal aspects, including controllability beyond text.
Method
The method adds scene conditioning, domain-specific tokenization for faces and salient objects, and classifier-free guidance adapted for transformers.
Results
The model achieves state-of-the-art FID and human-evaluation results and generates high-fidelity images at 512 × 512 pixels.
Takeaways & Limitations
Scene controllability supports complex scene generation, scene editing, anchored text editing, out-of-distribution generation, and story illustration.
Takeaways & Limitations
Because text-to-image generation lacks absolute ground truths, evaluation relies on image-distribution metrics such as FID rather than exact-image comparisons.
Abstract
from arXiv · showhide
Recent text-to-image generation methods provide a simple yet exciting conversion capability between text and image domains. While these methods have incrementally improved the generated image fidelity and text relevancy, several pivotal gaps remain unanswered, limiting applicability and quality. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene, (ii) introducing elements that substantially improve the tokenization process by employing domain-specific knowledge over key image regions (faces and salient objects), and (iii) adapting classifier-free guidance for the transformer use case. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels, significantly improving visual quality. Through scene controllability, we introduce several new capabilities: (i) Scene editing, (ii) text editing with anchor scenes, (iii) overcoming out-of-distribution text prompts, and (iv) story illustration generation, as demonstrated in the story we wrote.
1. Introduction
Existing text-to-image methods remain limited in controllability, human-perception alignment, quality, and resolution. The proposed method adds scene-based control, region-specific perceptual losses, and transformer classifier-free guidance, enabling new editing and generation capabilities.
- Limitations: Most current models accept only text, leaving structure, form, and arrangement loosely controlled.This restricts user influence over image content and context.
- Limitations: Generation losses often ignore human attention, especially the importance of faces, animals, and salient objects.The paper identifies a mismatch between face-pixel importance to people and whole-image losses.
- Limitations: Previous state-of-the-art methods were limited to 256 × 256 outputs, while super-resolution alternatives produced less favorable visual and quantitative results.The paper links 512 × 512 generation to the need for substantially higher quality and fewer artifacts.
- Method: The method introduces segmentation-derived scene tokens as implicit conditioning alongside text and image tokens.At inference, scene tokens can be generated by the transformer or extracted from an input image.
- Capabilities: The paper demonstrates complex scene generation, out-of-distribution generation, scene editing, and text editing with anchored scenes.It also uses controllability to support storytelling and story illustration.
- Method: Modified VQ-VAEs apply explicit losses to faces and salient objects, incorporating domain-specific perceptual knowledge through feature matching.These losses emphasize regions associated with human perception and attention.
- Method: Transformer classifier-free guidance replaces post-generation filtering with conditional and unconditional token streams, producing faster, higher-quality, better text-aligned results.The unconditional stream is trained by randomly replacing prompts with padding tokens.
2. Related Work
Related work spans generative image models, discrete image tokenization, scene-conditioned synthesis, and text-to-image generation. The paper builds on these lines by combining an autoregressive transformer with scene control, improved tokenization, and transformer-adapted classifier-free guidance.
- Generative image models: GANs and VAEs established adversarial and likelihood-based approaches for generating high-quality images.These model families form the broader generative-image background for the paper.
- Discrete representations: Discrete-representation models typically tokenize images first, then generate images in the resulting discrete latent space.VQ-VAE learns such representations through online clustering, while VQ-VAE-2 uses multiple scales for faster generation.
- Discrete representations: VQGAN improves VQ-VAE reconstruction with adversarial and perceptual losses, which this paper modifies to target faces and salient objects.The modification applies perceptual losses to specific image regions.
- Scene-conditioned synthesis: Segmentation-map image synthesis provides conditional control over desired outputs through supervised or unsupervised domain mappings.The related methods include CycleGAN, UNIT, pix2pix, and pix2pixHD.
- Text-to-image generation: DALL-E and CogView use autoregressive transformers over text and image tokens, while GLIDE uses image-conditioned diffusion and guided inference.LAFITE instead aligns text and images through a pretrained CLIP model.
- Text-to-image generation: This paper extends transformer text-to-image generation with scene tokens, improved tokenization, and classifier-free guidance.These are identified as its main contributions within the related-work discussion.
3. Method
The method combines scene conditioning with human-aware tokenization and an autoregressive transformer to improve controllability, structural consistency, and image quality. It supports scene-based generation and editing while using explicit perceptual losses and transformer classifier-free guidance.
- 3.1. Scene representation and tokenization: Scene conditioning uses panoptic, human, and face semantic segmentation groups to provide global layout context for image generation.The combined scene representation is encoded with VQ-SEG and can condition the image-generation process.
- 3.1. Scene representation and tokenization: Scene sketches enable generation of unusual objects and scenarios that text-only methods may struggle to represent.The paper demonstrates this capability for out-of-distribution prompts and scenarios.
- 3.1. Scene representation and tokenization: Edited semantic scenes support image regeneration with replaced or added classes, producing new context or content.The method extracts segmentations from an input image, edits the scene, and generates an image conditioned on the input text.
- 3.2. Adhering to human emphasis in the token space: Face-aware and object-aware feature-matching losses emphasize perceptually important regions during VQ image reconstruction.Face crops use a pretrained face-embedding network, while object crops use a pretrained VGG network; weighted segmentation loss also addresses face-part reconstruction.
- 3.6. Scene-based transformer: The method uses an autoregressive transformer over consecutive text, scene, and image token spaces encoded by BPE, VQ-SEG, and VQ-IMG.The complete token sequence is formed from text, scene, and image inputs before transformer training.
- 3.7. Transformer classifier-free guidance: Transformer classifier-free guidance trains with randomly omitted text and samples conditional and unconditional token streams in parallel.Guidance is applied to transformer logit scores during inference.
4. Experiments
Experiments evaluate Make-A-Scene with human judgments and FID across model configurations and datasets. The method achieves the strongest reported results in both evaluation settings while supporting 256 × 256 and 512 × 512 generation.
- Experimental setup: Experiments use a 4-billion-parameter transformer with 256 text tokens, 256 scene tokens, and 1024 image tokens.Depending on the model, decoded outputs have resolutions of 256 × 256 or 512 × 512 pixels.
- Experimental setup: The scene-based transformer is trained on 35 million text-image pairs assembled from CC12m, CC, YFCC100m subsets, and Redcaps.VQ-SEG and VQ-IMG are trained on CC12m, CC, and MS-COCO; MS-COCO is used unless otherwise specified.
- Evaluation protocol: Evaluation treats human judgments as the highest authority and uses FID as a secondary metric because text-to-image generation lacks absolute ground-truth images.FID evaluates distributions of generated images rather than correspondence to one specific target image.
- Evaluation protocol: The study compares the method with several state-of-the-art systems using FID and human evaluators when possible.A 4-billion-parameter DALL-E reimplementation enables a comparison using identical VQGAN tokenization.
- Human evaluation results: Human evaluators preferred the method over DALL-E and CogView across image quality, photorealism, and text alignment.Each comparison used 500 image pairs, with five evaluators per question and majority-vote percentages reported.
- FID comparison: The method achieves the lowest FID among evaluated models both with and without MS-COCO training data.FID is computed over 30k images generated from MS-COCO validation prompts without re-ranking.
4.7. Generating out of distribution
Scene conditioning enables generation beyond common text distributions, while supporting scene editing, text editing with anchored scenes, and consistent high-fidelity outputs at multiple resolutions.
- 4.7. Generating out of distribution: Scene conditioning supports unusual objects and scenarios, including cases where scene categories approximate nonexistent objects.The method uses simple sketches and nearby categories to address the category gap in out-of-distribution prompts.
- 4.7. Generating out of distribution: The model demonstrates high-fidelity text-to-image generation with scene-wise control and consistency between generations at 256 × 256 and 512 × 512 resolutions.Samples are reported for both model resolutions.
- 4.7. Generating out of distribution: Table 1 compares FID and human preference using 30k MS-COCO validation-prompt images, with results reported against prior models and a practical ground-truth bound.Human evaluation is expressed as the percentage of majority votes favoring the proposed method.
- 4.7. Generating out of distribution: Scene editing regenerates images according to edited semantic layouts, including replacing categories or adding a new object without changing the text.Examples replace sky and tree with sea, grass with sand, or add a giant dog sketch.
- 4.7. Generating out of distribution: Anchored scenes support new interpretations of existing images by combining an extracted semantic segmentation with edited text.The image is regenerated conditioned on the input scene and edited text.
- 4.7. Generating out of distribution: Story illustrations use simple sketches to translate ideas into paintings or realistic images while reducing text-to-image randomness and improving consistency between generations.The paper demonstrates this process through a children’s story.
5. Conclusion
The method moves text-to-image generation toward a more interactive experience by increasing control over outputs and improving perceptually important image regions.
- The method targets more interactive text-to-image generation, giving people greater control over generated outputs for applications such as storytelling.
- Improving faces and salient objects increases the method’s favorability in human evaluations and objective metrics.
A.1. VQ-SEG
VQ-SEG uses segmentation categories spanning panoptic segmentation, human parsing, and face parsing, with specified training and dictionary settings.
- VQ-SEG is trained for 600k iterations with batch size 48 and dictionary size 1024.
- Its segmentation groups contain 133 panoptic, 20 human-parsing, and 5 face-parsing categories.
- Face-part categories include eyebrows, eyes, nose, outer-mouth, and inner-mouth.
A.2. VQ-IMG
The VQ-IMG models use different training durations and batch sizes while sharing a dictionary size, with face-aware loss normalization specified across feature-map resolutions.
- VQ-IMG256 and VQ-IMG512 are trained for 800k and 940k iterations, respectively, with batch sizes of 192 and 128.
- Both VQ-IMG models use dictionary size 8192 and channel multipliers beginning with [2].
- The final face-aware loss setting uses αf1 = 0.1 and αf2 = 0.25, with per-layer values assigned across resolutions from 1×1 to 128×128.
- The object-aware loss normalization values are taken from prior work based on LPIPS.
A.3. Scene-based transformer
The 256 × 256 and 512 × 512 models share the same transformer architecture and training schedule, while inference samples tokens through top-half logit selection and multinomial sampling.
- Both transformers use 48 layers, 48 attention heads, and an embedding dimension of 2560.
- The models were trained for 170k iterations with batch size 1024, Adam optimization, and a 4.5 × 10^-4 starting learning rate.
- At each inference step, the method retains the half of logits with highest probabilities before applying softmax.
- A single token is then sampled from the resulting multinomial probability distribution.
B. Additional samples
Additional examples evaluate Make-A-Scene on challenging text prompts and text-plus-segmentation inputs, including scenarios requiring approximate category substitutions.
- Text and segmentation inputs: Figures 9–12 provide additional samples generated from text and segmentation inputs.
- Additional samples: Different text colors emphasize the many objects and scenarios being attended during generation.
- Category substitutions: Because octopus and dinosaur categories are unavailable, the examples use cat and giraffe categories instead.
- Challenging text inputs: Figures 7–8 provide additional samples generated from challenging text inputs.