PaperlayerView exact revision on arXiv

Source-linked AI summary

Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

Wan-Cyuan Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, Yu-Chiang Frank Wang

arXiv:2208.13753v2cs.CVcs.AIcs.LG

TL;DR

Complex-scene synthesis requires models to capture both global structures and object details, which existing diffusion approaches may not represent well at a single scale. Frido addresses this with multi-scale coarse-to-fine diffusion over vector-quantized features and supports multiple conditioning modalities. It reports five new state-of-the-art results and improved inference efficiency, while noting unresolved questions about feature normalization and which semantic knowledge best supports coarse-to-fine learning.

  • Problem

    Existing diffusion models may struggle with complex scenes and highly abstractive conditions because single-scale representations do not adequately capture composition and high-level semantic relations.

  • Method

    Frido uses multi-scale vector-quantized features, a feature-pyramid U-Net, and coarse-to-fine modulation to denoise images from global structures to fine details.

  • Results

    Frido achieves 5 new state-of-the-art results across diverse cross-modal image-synthesis tasks and mitigates diffusion models’ slow inference.

  • Takeaways & Limitations

    Frido provides a generic coarse-to-fine diffusion framework for conditional image synthesis with text, layouts, scene graphs, and labels.

  • Takeaways & Limitations

    Frido leaves open how feature normalization and the specific high- or low-level knowledge encoded in features affect coarse-to-fine diffusion learning.

Abstract

from arXiv · show

Diffusion models (DMs) have shown great potential for high-quality image synthesis. However, when it comes to producing images with complex scenes, how to properly describe both image global structures and object details remains a challenging task. In this paper, we present Frido, a Feature Pyramid Diffusion model performing a multi-scale coarse-to-fine denoising process for image synthesis. Our model decomposes an input image into scale-dependent vector quantized features, followed by a coarse-to-fine gating for producing image output. During the above multi-scale representation learning stage, additional input conditions like text, scene graph, or image layout can be further exploited. Thus, Frido can be also applied for conditional or cross-modality image synthesis. We conduct extensive experiments over various unconditioned and conditional image generation tasks, ranging from text-to-image synthesis, layout-to-image, scene-graph-to-image, to label-to-image. More specifically, we achieved state-of-the-art FID scores on five benchmarks, namely layout-to-image on COCO and OpenImages, scene-graph-to-image on COCO and Visual Genome, and label-to-image on COCO. Code is available at https://github.com/davidhalladay/Frido.

1 Introduction

Frido targets complex, condition-rich image synthesis, where existing diffusion models struggle to represent global scene structure, semantic relations, and object details efficiently. It introduces multi-scale coarse-to-fine diffusion with feature-pyramid representations and reports broad conditional-generation results, including five new state-of-the-art settings.

  • Motivation: Existing diffusion models can struggle with complex scenes and abstractive conditions because single-scale processing may miss object composition and high-level semantic relations.The paper also identifies iterative denoising as computationally expensive, especially for high-resolution outputs.
  • Frido: Frido uses MS-VQGAN to encode images into multi-scale discrete visual features and a feature-pyramid U-Net to denoise them coarse-to-fine from global structure to fine details.Its coarse-to-fine modulation conditions lower-level denoising on already generated higher-level features.
  • Applications: Frido supports diverse conditions, including text, box layouts, scene graphs, and labels, enabling cross-modal image synthesis across complex-scene tasks.The framework is presented as generic rather than limited to one conditioning modality.
  • Results: Frido achieves 5 new state-of-the-art results across layout-to-image, scene-graph-to-image, and label-to-image benchmarks.The reported settings cover COCO, OpenImages, and Visual Genome.
  • Efficiency: Frido also mitigates diffusion models’ slow inference while introducing minimal extra parameters.The paper highlights a head-to-head speed comparison with LDM.

2 Preliminary

The preliminary section reviews diffusion as iterative forward corruption and learned backward denoising, then explains latent diffusion’s lower-dimensional representation. Latent codes reduce computational cost because denoising operates at substantially lower spatial resolution than image pixels.

  • Diffusion Models: Diffusion models gradually transform data into approximately standard Gaussian noise, then learn a reverse denoising process to reconstruct samples.The denoising network predicts the noise applied at each step, and synthesis runs this process iteratively from random noise.
  • Latent Diffusion Models: Latent Diffusion Models replace high-dimensional image pixels with low-dimensional codes produced by a pretrained VQGAN or KL-autoencoder.For an image x0, an encoder produces latent codes z0 that are downsampled relative to the original spatial dimensions.
  • Latent Diffusion Models: Latent diffusion greatly reduces computation because diffusion and denoising operate on lower-resolution latent representations, while a decoder reconstructs the final image.The autoencoding step is performed once per image, whereas the iterative process uses the lower-resolution latent.

3 Methodology

Frido models complex-scene images through multi-scale latent representations and coarse-to-fine diffusion, combining global structures with object details. Its MS-VQGAN provides scale-dependent features, while PyU-Net denoises them sequentially with shared, modulated computations.

  • 3.1 Learning Multi-Scale Perceptual Latents: MS-VQGAN encodes images into multi-scale latent feature maps, encouraging low-level details at z1 and high-level shapes and structures at zN.The features are quantized, fused, upsampled, concatenated, and decoded for image reconstruction.
  • 3.1 Learning Multi-Scale Perceptual Latents: MS-VQGAN is designed to preserve structure and detail while representing different semantic levels across its multi-scale codes.
  • 3.2 Feature Pyramid Latent Diffusion Model: Frido diffuses feature scales sequentially rather than simultaneously, progressing from low-level details toward high-level structure across N × T timesteps.This process corrupts object details, object shape, and finally the entire-image structure, enabling multiple semantic levels.
  • 3.2 Feature Pyramid Latent Diffusion Model: PyU-Net denoises features from high-level to low-level scales using one shared U-Net with lightweight level-specific projections, reducing trainable parameters.The shared U-Net is reused across levels after projecting features into a shared space.
  • 3.2 Feature Pyramid Latent Diffusion Model: Coarse-to-fine modulation conditions low-level denoising on generated high-level features while incorporating stage and timestep information.The design applies feature-based modulation followed by stage-temporal modulation, with convolution and SiLU layers between operations.
  • 3.2 Feature Pyramid Latent Diffusion Model: Frido adds only a moderate parameter increase and performs diffusion on lower-resolution maps first, supporting faster inference than conventional diffusion processing.The authors report that this design achieves strong results under a similar compute budget to a fast diffusion model.

4 Experiments

Frido is evaluated across diverse conditional image-generation tasks and achieves strong quality, semantic consistency, and efficiency on complex scenes. The experiments analyze datasets, conditional-generation performance, architectural components, and speed-quality trade-offs.

  • 4.1 Datasets and Evaluation: The evaluation covers T2I, SG2I, Label2I, and Layout2I across COCO, Visual Genome, and OpenImages using FID, IS, and task-specific metrics.Additional experiments include user preference studies and unconditional image generation, with detailed settings reported in the supplementary material.
  • 4.2 Conditional Complex Scene Generation: For COCO text-to-image generation, Frido improves over LDM by approximately 2 FID points and 1 CLIP-Score point, with reported FID 15.38 versus 11.24 and CLIP-Score 0.6607 versus 0.7046.A test-time CLIP ranking variant further improves the metrics and reaches results comparable to LAFITE.
  • 4.2 Conditional Complex Scene Generation: Frido outperforms previous methods on COCO and Visual Genome scene-graph generation in FID and IS, with CLIP-score gains of ≈2% on COCO and ≈0.2% on Visual Genome.The results are attributed to improved modeling of complex relations using feature pyramids and coarse-to-fine generation.
  • 4.2 Conditional Complex Scene Generation: In COCO label-to-image generation, Frido surpasses prior approaches on FID, precision, and recall for 3-8 labels and retains state-of-the-art FID for 2-30 labels.The evaluation targets image quality and object-level diversity under multi-object conditioning.
  • 4.2 Conditional Complex Scene Generation: Frido outperforms prior layout-to-image methods by at least 2 FID points on both COCO and OpenImages, while also achieving the best YOLO scores and sceneFID.These metrics indicate stronger reported visual realism for instance-level objects under layout conditioning.
  • 4.3 Model Analysis: PyU-Net reduces parameters from 1.18B to 590M, while adding coarse-to-fine modulation raises the total to 697M and further improves all metrics over LDM baselines.The ablation evaluates T2I on COCO and SG2I on Visual Genome with bootstrap confidence intervals.
  • 4.3 Model Analysis: At similar inference budgets, Frido provides a performance gain over LDM by shifting part of denoising to lower-resolution features.The speed-quality analysis varies inference timesteps and compares FID against per-image inference cost on a V100.
  • 4.2 Conditional Complex Scene Generation: Frido achieves state-of-the-art FID scores in five settings spanning scene-graph-to-image, label-to-image, and layout-to-image generation.The reported settings cover COCO and Visual Genome for scene graphs, COCO for labels, and COCO and OpenImages for layouts.

5 Related Work

Related work spans major generative-model families, two-stage latent models, and coarse-to-fine image-generation methods. Frido combines VAE-style representation learning with diffusion and uses shared multi-scale networks to reduce overhead.

  • More Generative Models for Image Synthesis: GANs, VAEs, autoregressive models, flow-based methods, and diffusion models have all contributed to progress in photo-realistic image synthesis.Frido is described as a hybrid of the VAE and diffusion-model families, while remaining orthogonal to large-scale text-to-image pre-training.
  • Two-Stage Generative Models: Two-stage generative models first encode images into lower-resolution discrete latent spaces and then model those representations with autoregressive or diffusion networks.Frido contributes MS-VQGAN and PyU-Net to the representation-learning and diffusion stages, respectively.
  • Coarse-to-Fine Image Generation Approaches: Coarse-to-fine methods generate images across resolutions or information levels, while Frido shares its core network across scales to minimize overhead relative to single-scale models.Prior examples include AttnGAN and StackGAN, which progressively upscale low-resolution images.

6 Conclusion

Frido uses a coarse-to-fine diffusion framework to generate images from multi-scale features, aiming to preserve global structure and object detail while supporting diverse inputs. Its sequential multi-scale diffusion and denoising processes operate from fine details toward structure during corruption and from coarse structure toward details during generation.

  • 6 Conclusion: Frido achieves 5 new state-of-the-art results across cross-modal image synthesis tasks while mitigating diffusion models’ slow inference.The model applies to diverse conditional inputs and uses an under-explored coarse-to-fine prior.
  • 6 Conclusion: Frido introduces a feature pyramid U-Net with coarse-to-fine modulation and multi-scale codebooks produced by MS-VQGAN.The shared U-Net and modulation mechanism are designed to denoise visual features across spatial scales in a top-down fashion.
  • A.1 Formulation of the Diffusion and Denoising Process of Frido: The diffusion process sequentially adds Gaussian noise across feature-map scales, progressing from low-level features to high-level features and corrupting details before overall structure.This ordering is intended to capture information at different semantic levels.
  • A.1 Formulation of the Diffusion and Denoising Process of Frido: The denoising process initializes all scales from a normal distribution and generates them sequentially from high-level to low-level features for coarse-to-fine synthesis.Each lower-scale denoising step conditions on previously produced higher-scale feature maps.
  • A.1 Formulation of the Diffusion and Denoising Process of Frido: During testing, sampled multi-scale latent features are decoded back into image pixels with a pre-trained decoder.The denoising autoencoder predicts the added noise variant during the reverse process.
  • B Implementation Details: The implementation details specify training and testing environments, hyperparameters, and multi-modal condition encoding for reproducibility.The appendix covers the operational setup and task-specific implementation choices.

B.1 Training/Testing Environment

The appendix fixes a reproducible training setup and describes task-specific hyperparameters and encodings for text, scene graphs, labels, and layouts.

  • B.1 Training/Testing Environment: Models use Linux with 8 Nvidia Tesla V100 GPUs, PyTorch Lightning 1.4.2, float32 precision, AdamW, fixed seed 23, and no learning-rate scheduler unless specified.The optimizer uses betas=(0.9, 0.999), eps=1e−08, and weight decay=0.01.
  • B.2 Hyper-Parameter Settings: Hyperparameter settings are organized separately for conditional tasks, unconditional generation, and model analysis.The corresponding settings are provided in Tables A, B, and C.
  • B.3 Encoding Multi-Modal Conditions: Text conditions are tokenized and encoded into embeddings with a BERT tokenizer and transformer module.This encoding converts captions into sequences of condition embeddings.
  • B.3 Encoding Multi-Modal Conditions: Scene graphs are encoded from word, part-of-pair, and object-ID token sequences whose BERT embeddings are summed.The procedure follows the LT-Net representation strategy.
  • B.3 Encoding Multi-Modal Conditions: Image-level labels are mapped to category names, concatenated into text, and encoded like text-to-image conditions.The label representation therefore uses the text-conditioning pathway.
  • B.3 Encoding Multi-Modal Conditions: Layout conditions are discretized into tokenized bounding-box triples encoding position, size, and category before BERT encoding.Each box is represented as an (l, b, c)-tuple for top-left position, bottom-right position, and category.

C.1 Evaluation Metrics

The evaluation uses complementary metrics for image quality, semantic alignment, object fidelity, distribution coverage, and multi-scale reconstruction quality.

  • C.1 Evaluation Metrics: FID, IS, and SceneFID measure overall image quality, classifier-based image distributions, and instance-level fidelity, respectively.SceneFID compares generated and ground-truth instances cropped by corresponding bounding boxes.
  • C.1 Evaluation Metrics: CLIPscore evaluates semantic compatibility between generated images and text or scene-graph conditions using a reference-free CLIP-based measure.The implementation uses CLIP pretrained on 400M image-caption pairs and officially released code.
  • C.1 Evaluation Metrics: YOLO detection AP at a 0.50 threshold measures object alignment and fidelity for layout-to-image generation.Evaluation uses COCO-pretrained YOLOv4 weights.
  • C.1 Evaluation Metrics: Precision and recall assess whether generated samples are realistic relative to the data distribution and how much of that distribution the model covers.Precision reflects support overlap for generated samples, whereas recall reflects coverage of the real distribution.
  • C.1 Evaluation Metrics: PSNR and SSIM evaluate reconstruction quality in the multi-scale vector-quantization analysis.These metrics assess similarity between reconstructed and reference images.

C.2 Datasets and Evaluation Setting

Experiments cover unconditional and conditional generation across multiple datasets and standardized evaluation settings. Additional layout-to-image results report that Frido surpasses prior approaches on FID and Scene FID.

  • C.2 Datasets and Evaluation Setting: The evaluation setting covers text-to-image on COCO 2014, scene-graph-to-image on COCO and Visual Genome, label-to-image on COCO, and layout-to-image on COCO and OpenImage.Each task follows established dataset splits or protocols from prior work.
  • C.2 Datasets and Evaluation Setting: Text-to-image models train on 83K COCO 2014 captioning images and are evaluated on the full 41K-image validation set.Scene-graph-to-image uses 25K/2K COCO train/test images and 62K/5K Visual Genome train/test data.
  • C.2 Datasets and Evaluation Setting: Label-to-image evaluation uses COCO 2017 Panoptic with 118K/5K training/validation images and validation subsets containing either 3–8 or 2–30 objects.The two subsets contain 3,276 and 4,722 images, respectively.
  • C.2 Datasets and Evaluation Setting: Layout-to-image evaluation reports FID and IS on 2,048 COCO testing images and 2,048 sampled OpenImage validation images.The training setup follows the COCO 2017 Stuff Segmentation Challenge split.
  • C.2 Datasets and Evaluation Setting: Unconditional generation is evaluated on LSUN-bed, CelebA-HQ, and Landscape using 50K random samples against each dataset’s training set.FID and precision/recall are computed with torch-fidelity and the ADM script.
  • D.1 Conditional Image Generation: Qualitative comparisons include text-to-image, scene-graph-to-image, label-to-image, and layout-to-image generation under the same settings as the main experiments.The supplementary figures provide additional examples for each conditional task.
  • D.1 Conditional Image Generation: Frido again surpasses previous approaches on both FID and Scene FID for layout-to-image generation on COCO-stuff 2017 and Visual Genome.The comparison includes HCSS, LAMA, Context-L2I, and TwFA and supports Frido’s generation of multiple objects’ shapes and details.

D.2 User Preference Studies

User studies and unconditional-generation experiments evaluate Frido’s perceptual quality, condition relevance, and generalization across datasets. Frido shows stronger condition relevance and high data-manifold coverage while maintaining comparable image-quality metrics.

  • D.2 User Preference Studies: Frido achieved higher condition relevance than previous approaches in user studies for text-to-image and layout-to-image generation.The study collected 1,320 answers from 22 users comparing Frido with LDM, VQ-Diffusion, and Lafite.
  • D.2 User Preference Studies: Frido surpassed VQ-Diffusion and Lafite on text-to-image fidelity and LDM on layout-to-image fidelity, although LDM had higher visual preference on text-to-image.The authors attribute LDM’s visual preference to smoother images despite lower accordance with constraints.
  • D.3 Unconditional Image Generation: Frido achieved comparable FID and precision while obtaining high recall across CelebA-HQ, LSUN-bed, and Landscape unconditional-generation benchmarks.The experiments used FID, Precision, and Recall to assess image quality and coverage of the modeled data manifold.
  • D.3 Unconditional Image Generation: The unconditional experiments attribute Frido’s high data-manifold coverage to its feature-pyramid design and coarse-to-fine learning strategy.The authors report that this design allows the diffusion model to learn dataset distributions more easily.

D.4 Model Analysis

Model analysis examines quantization quality, feature-scale choices, noise augmentation, and computational cost. The results show improved visual-detail preservation, a cost–performance trade-off across scales, robust noise augmentation at α = 0.1, and comparable inference cost to LDM.

  • MS-VQGAN Performance: MS-VQGAN outperformed prior quantization models on PSNR and SSIM while achieving comparable reconstruction FID and IS.The comparison used equal codebook sizes and evaluated reconstruction fidelity and visual-detail preservation.
  • Feature Scales of Frido: In N = 2 settings, Frido-f8f4 improves FID from 40.14 to 38.68 relative to Frido-f16f8, while increasing inference time by approximately four times, demonstrating a resolution–efficiency trade-off.
  • Feature Scales of Frido: Adding the f32 feature decreased performance because weaker high-level reconstruction caused early coarse-to-fine errors to accumulate at later denoising stages.The authors therefore adopted N = 2, using Frido-f16f8 and Frido-f8f4, for all tasks.
  • Noise Augmentation: Noise augmentation was most effective at the default ratio α = 0.1, whereas α = 0 and α = 0.5 degraded performance through train–test mismatch or excessive feature noise.Without augmentation, T2I FID worsened from 106 to 122; the authors report α = 0.1 as robust.
  • Computation Cost: Frido achieved comparable per-image inference cost to LDM at 200 timesteps while improving performance with 697M total parameters.The comparison considered FLOPs, parameter counts, and inference time on the same timestep setting.

E.1 Limitations and Future Directions

The paper identifies unresolved representation-learning questions and ethical risks surrounding Frido. Future work targets regularization and explicit control of multiscale information, while deployment must consider malicious use and fairness concerns.

  • E.1 Limitations and Future Directions: MS-VQGAN features are not regularized across scales, producing varying means and standard deviations that can impair diffusion training.The authors rescale each feature by its reciprocal standard deviation but argue that a dedicated latent regularization objective is still needed.
  • E.1 Limitations and Future Directions: Future work should determine what high- and low-level information best supports coarse-to-fine diffusion learning.The paper proposes guiding multiscale quantization with objectives such as reconstructing low-resolution inputs from high-level features alone.
  • E.2 Ethics Concerns: Frido’s ability to generate realistic images from labels, language, layouts, and scene graphs creates a vulnerability to malicious use.The concern applies to the model’s broad cross-modal conditioning capabilities.
  • E.2 Ethics Concerns: Fairness in generating categories such as humans remains a potential issue, which the authors associate with bias in training-data collection.The paper points to prior work discussing these social issues and possible mitigation strategies.
Loading 2208.13753v2…