Source-linked AI summary
Bigger is not Always Better: Scaling Properties of Latent Diffusion Models
Kangfu Mei, Zhengzhong Tu, Mauricio Delbracio, Hossein Talebi, Vishal M. Patel, Peyman Milanfar
TL;DR
Latent diffusion models offer strong generation quality but are costly because they require multi-step sampling, and the role of model size in sampling efficiency has been insufficiently examined. The paper empirically scales LDMs from 39M to 5B parameters across text-to-image and downstream tasks, finding that smaller models frequently outperform larger ones under identical sampling costs. The trend persists across samplers, downstream settings, and distilled models, while the conclusions are limited to the studied model family and metric-based evaluation.
Problem
The paper addresses limited evidence about how LDM model size affects sampling efficiency, an important issue because multi-step sampling creates high deployment latency.
Method
The authors train and finetune scaled LDMs ranging from 39M to 5B parameters, evaluating sampling efficiency across text-to-image, downstream tasks, samplers, training compute, and distilled models.
Results
Under identical sampling costs, smaller models frequently outperform larger models, with the scaling-efficiency pattern remaining consistent across multiple samplers and distilled models.
Takeaways & Limitations
Model-size scaling provides a promising direction for improving LDM generation under limited inference budgets.
Takeaways & Limitations
The claims are specific to the studied LDM family, and the evaluation relies on visual inspection plus FID and CLIP rather than human evaluations.
Abstract
from arXiv · showhide
We study the scaling properties of latent diffusion models (LDMs) with an emphasis on their sampling efficiency. While improved network architecture and inference algorithms have shown to effectively boost sampling efficiency of diffusion models, the role of model size -- a critical determinant of sampling efficiency -- has not been thoroughly examined. Through empirical analysis of established text-to-image diffusion models, we conduct an in-depth investigation into how model size influences sampling efficiency across varying sampling steps. Our findings unveil a surprising trend: when operating under a given inference budget, smaller models frequently outperform their larger equivalents in generating high-quality results. Moreover, we extend our study to demonstrate the generalizability of the these findings by applying various diffusion samplers, exploring diverse downstream tasks, evaluating post-distilled models, as well as comparing performance relative to training compute. These findings open up new pathways for the development of LDM scaling strategies which can be employed to enhance generative capabilities within limited inference budgets.
1 Introduction
The paper examines whether latent diffusion model size affects sampling efficiency, addressing a gap left by architecture and sampler-focused efficiency work. Across text-to-image and downstream settings, it finds that smaller models can outperform larger ones under constrained sampling budgets, while scaling remains beneficial when computation is less constrained.
- 1 Introduction: The work targets low sampling efficiency, where multi-step diffusion makes total sampling cost the product of sampling steps and per-step computation.The paper contrasts this iterative cost with single-shot generative models and motivates efficiency optimization for deployment.
- 1 Introduction: Prior efficiency efforts mainly accelerate individual network steps, reduce sampling steps, or distill multi-step behavior, leaving smaller and less redundant models comparatively underexplored.The paper positions model-size scaling as a complementary efficiency axis.
- 1 Introduction: The study trains 12 text-to-image LDMs from 39 million to 5 billion parameters and evaluates how model size affects sampling efficiency across downstream tasks.The downstream evaluation includes real-world super-resolution and subject-driven text-to-image synthesis.
- 1.1 Summary: Pretraining performance scales with training compute, while downstream performance scales with pretraining quality and cannot be fully matched by smaller models through extra training alone.Together, these findings separate the benefits of more training compute from those of a stronger pretrained model.
- 1.1 Summary: Smaller models initially outperform larger models in image quality under a fixed sampling budget, whereas larger models recover an advantage for detail generation when computational constraints are relaxed.This establishes the section’s central scaling result and its dependence on the inference budget.
- 1.1 Summary: Smaller models retain sampling-efficiency advantages across DDIM, DDPM, and DPM-Solver++, downstream tasks below 20 steps, and distilled models.These findings indicate that the reported trend is not limited to one sampler, task setting, or undistilled model family.
2 Related Work
Related work establishes that larger diffusion models can improve quality, but their inference cost motivates examining efficient alternatives. The paper’s scaled LDM family changes denoising U-Net width while holding other architectural elements consistent.
- Scaling laws: The paper relates diffusion scaling to broader generative-model scaling work, including language-model studies that predict performance from model size.This places LDM scaling within a wider investigation of how capacity and compute affect generative performance.
- Efficient diffusion models: Prior diffusion work reports improved generative performance with increasing model size, while billion-parameter LDMs can exceed practical inference budgets.This motivates studying whether smaller models provide better quality-efficiency trade-offs.
- Efficient non-diffusion generative models: Other generative-model families can require fewer sampling steps than diffusion models, but some use substantially larger models, including 4 billion parameters.This comparison highlights that sampling-step efficiency and parameter efficiency are distinct considerations.
- Evaluation setup: Table 1 evaluates FID and CLIP on 30k COCO-2014 samples using 50-step DDIM and CFG 7.5, reporting denoising-U-Net costs rather than text-encoder or latent autoencoder costs.The reported model, training, and inference costs exclude the 1.4B text encoder and 250M latent encoder and decoder.
3 Scaling LDMs
The paper builds a controlled family of LDMs and measures how model size and training compute affect text-to-image and downstream performance. Results show smooth compute-related scaling in pretraining, while downstream quality and detail recovery often favor larger pretrained models.
- Scaling architecture: The family is constructed by incrementally increasing residual-block filters while preserving other architectural elements, using WebLI training data and 500K DDIM training steps.The controlled construction supports comparisons across model size and training compute.
- 3.2 Pretraining scales downstream performance: In 4× real-image super-resolution, model size improves FID and visual detail recovery, whereas training compute has the strongest effect on LPIPS.The divergence between metrics is accompanied by visual evidence that larger models recover finer details more effectively.
- Evaluation setup: The experiments evaluate scaled models on COCO-2014 text-to-image validation data and DIV2K patches degraded with RealESRGAN for real-world super-resolution.The evaluation uses 30k text-to-image samples and 3k randomly cropped super-resolution patches.
- 3.1 Training compute scales text-to-image performance: Generative performance follows similar trends across model sizes relative to training compute, especially after training stabilizes around 200K iterations.Training compute is defined as relative per-model cost multiplied by training iterations.
- 3.2 Pretraining scales downstream performance: Visual quality in 4× super-resolution directly improves with model size, and differences in pretraining performance transfer to downstream results quantitatively and qualitatively.The same pretrained scaled models are compared under 50-step DDIM sampling.
3.2 Pretraining scales downstream performance
Pretraining scale improves downstream visual quality, but the sampling-efficiency comparison depends on budget: larger models recover finer details, whereas smaller models can achieve better FID at limited sampling cost.
- 3.2 Pretraining scales downstream performance: Larger pretrained models excel at recovering fine-grained details in super-resolution and show a similar visual-quality advantage in DreamBooth.The super-resolution comparison reports larger models outperforming smaller ones on fine details, while DreamBooth results show two quality tiers favoring larger models.
- 3.2 Pretraining scales downstream performance: Pretraining performance has greater influence on super-resolution FID than the duration of downstream finetuning.Large super-resolution models achieve superior results even after short finetuning periods compared with smaller models.
- 3.3.1 Analyzing the effect of CFG rate: CFG-rate changes affect visual quality substantially, so the study selects an optimal CFG rate separately for each model size and sampling-step count.FID is used to determine optimal sampling performance because it directly measures visual quality, whereas CLIP emphasizes semantic similarity.
- 3.3.2 Scaling efficiency trends: Smaller models frequently achieve better FID than larger models at a fixed normalized sampling cost, and can match larger models visually under similar cost.The comparison uses normalized cost × sampling steps and selects optimal performance across CFG rates.
- 3.2 Pretraining scales downstream performance: As sampling steps increase beyond 20, performance differences between super-resolution model sizes become less pronounced.Within 20 or fewer steps, smaller models tend to achieve lower FID scores.
3.4 Scaling sampling-efficiency in different samplers
Sampling-efficiency trends persist across deterministic, stochastic, and higher-order samplers, although the samplers differ in image quality at low step counts and have different practical step ranges.
- 3.4 Scaling sampling-efficiency in different samplers: Smaller models tend to outperform larger models at the same sampling cost with DDPM, DDIM, and DPM-Solver++, showing sampler-invariant scaling efficiency.DPM-Solver++ testing was restricted to at most 20 steps because it is not designed for larger step counts.
- 3.4 Scaling sampling-efficiency in different samplers: DDPM typically produces lower-quality results than DDIM with fewer sampling steps, whereas DPM-Solver++ generally outperforms DDIM in image quality.These quality differences do not remove the observed advantage of smaller models under matched sampling cost.
- Downstream super-resolution: Super-resolution evaluation compares limited sampling of 20 or fewer steps with a larger-step regime, without classifier-free guidance.The study directly uses super-resolution sampling results and examines both fewer and more than 20 sampling steps.
- 3.6 Scaling sampling-efficiency in distilled LDMs: Diffusion distillation improves four-step text-to-image performance across model sizes and preserves the sampling-efficiency pattern.At normalized sampling cost approximately 8, the undistilled 83M model achieves performance similar to the distilled 866M model.
4 Conclusion
Across LDMs scaled from 39 million to 5 billion parameters, smaller models often outperform larger models at identical sampling cost, while sampling-efficiency trends remain consistent across samplers and distilled models.
- 4 Conclusion: Across model sizes from 39 million to 5 billion parameters, smaller LDMs frequently outperform larger ones under identical sampling costs.The models were trained from scratch on a web-scale text-to-image dataset and later finetuned for downstream tasks.
- 4 Conclusion: Sampling-efficiency trends remain consistent across stochastic and deterministic samplers and also hold for distilled models.The conclusion frames model-size selection as a way to balance performance and efficiency under practical inference constraints.
- Limitations and future work: The analysis is limited by avoiding human evaluations, so discrepancies between visual quality and FID or CLIP scores remain possible.The study used visual inspection together with established metrics because evaluating more than 1000 variants would require many human-evaluation combinations.
- Limitations and future work: Scalability claims are restricted to the studied LDM family and require testing on transformer-based, multimodal, and cascaded diffusion families.The paper identifies DiT, SiT, MM-DiT, DiS, Imagen3, and Stable Cascade as directions for future evaluation.
A Scaling the text-to-image performance
The supplementary visual comparisons use identical prompts and sampling parameters across scaled LDMs, illustrating text-to-image results from 83M to 5B parameters.
- A Scaling the text-to-image performance: Figures 14–16 compare 83M–5B LDMs using the same prompt, 50-step DDIM sampling, and CFG rate 7.5.Holding these settings fixed supports visual comparison across model sizes.
B Scaling downstream performance
Super-resolution quality depends on both training compute and pretraining quality: larger models improve visual quality, while pretraining boosts downstream capability.
- Additional visual comparisons and IS measurements supplement the main super-resolution evaluations.
- IS trends with training compute still depend on pretraining, mirroring the corresponding FID trends.
- Larger models improve visual quality in 4× super-resolution, and pretraining boosts their super-resolution capabilities.
C Scaling sampling-efficiency in distilled LDMs
Guided distillation does not accelerate different model sizes consistently, whereas visual comparisons assess distilled and undistilled models at matched sampling cost.
- Guided distillation produces inconsistent performance improvements across the 83M and 223M models.The comparison evaluates distillation stages s16 and s8 for each model size.
- Visual results compare CoDi-distilled and undistilled models under the same sampling cost to assess sampling efficiency.
D Scaling the sampling-efficiency
At approximately equal inference cost, smaller LDMs can match or surpass larger models in visual quality, demonstrating a sampling-efficiency advantage under constrained budgets.
- Additional visual comparisons across scaled models under uniform sampling cost reinforce that smaller models can match their larger counterparts.
- Smaller models can produce comparable or better visual results than larger models under similar sampling cost.The comparison uses model GFLOPs × sampling steps as the inference-cost measure.
E Scaling interpretability of text prompt interpolatation
Text-prompt interpolation evaluates interpretability across model sizes, with larger models producing more semantically coherent and visually plausible interpolations.
- Larger models generate more semantically coherent and visually plausible text-prompt interpolations than smaller models.The 2B model more accurately interprets interpolated prompts, including a tablet computer with a touch pen.
- Interpolation blends two prompt embeddings as αA + (1 − α)B, using a 20-step DDIM sampler and CFG of 7.5.