PaperlayerView exact revision on arXiv

Source-linked AI summary

When Do Diffusion Models learn to Generate Multiple Objects?

Yujin Jeong, Arnas Uselis, Iro Laina, Seong Joon Oh, Anna Rohrbach

arXiv:2605.00273v2cs.CVcs.AI

TL;DR

Multi-object generation remains unreliable in text-to-image diffusion models, and the role of imperfect training data is unclear. The paper introduces MOSAIC to isolate concept and compositional generalization, finding that scene complexity and held-out combinations drive failures, while counting is especially fragile with limited data.

  • Problem

    The paper asks how reliably diffusion models generate multi-object compositions under imperfect training-data distributions.

  • Method

    The authors introduce MOSAIC, a controlled dataset-generation framework isolating attribution, counting, and spatial relations across concept and compositional generalization regimes.

  • Results

    Diffusion models eventually generalize concepts with sufficient data, but scene complexity matters more than imbalance, counting is fragile in low-data regimes, and compositional generalization collapses as held-out combinations increase.

  • Takeaways & Limitations

    The findings indicate that current diffusion models lack mechanisms for robust multi-object compositional generation.

Abstract

from arXiv · show

Text-to-image diffusion models achieve impressive visual fidelity, yet they remain unreliable in multi-object generation. Despite extensive empirical evidence of these failures, the underlying causes remain unclear. We begin by asking how much of this limitation arises from the data itself. To disentangle data effects, we consider two regimes across different dataset sizes: (1) concept generalization, where each individual concept is observed during training under potentially imbalanced data distributions, and (2) compositional generalization, where specific combinations of concepts are systematically held out. To study these regimes, we introduce mosaic (Multi-Object Spatial relations, AttrIbution, Counting), a controlled framework for dataset generation. By training diffusion models on mosaic, we find that scene complexity plays a dominant role rather than concept imbalance, and that counting is uniquely difficult to learn in low-data regimes. Moreover, compositional generalization collapses as more concept combinations are held out during training. These findings highlight fundamental limitations of diffusion models and motivate stronger inductive biases and data design for robust multi-object compositional generation.

1. Introduction

The introduction frames multi-object generation as a major weakness of text-to-image diffusion models and investigates how training-data distributions and dataset size affect this limitation. It studies concept generalization and compositional generalization using the controlled MOSAIC dataset, finding that scene complexity and held-out combinations drive failures.

  • Motivation: Diffusion models achieve above 80% accuracy on single-object tasks but often fall below 50% on multi-object compositional benchmarks.These failures involve counting object instances, attributing attributes correctly, and preserving spatial relations.
  • Research questions: The study examines whether imperfect data distributions and dataset size limit reliable multi-object generation through concept and compositional generalization.Concept generalization tests learning individually observed concepts under imbalance, whereas compositional generalization tests recombination of known concepts into unseen combinations.
  • Method: MOSAIC parameterizes object counts, color attribution, and spatial relations separately to diagnose the causal effects of dataset properties.The authors train two diffusion architectures without additional inductive biases such as layout conditioning.
  • Findings: Concepts generalize reliably in multi-object scenarios once the dataset is sufficiently large, while low-data performance is degraded more by scene complexity than concept imbalance, especially for Counting.The introduction identifies Counting as particularly difficult in low-data regimes.
  • Findings: Compositional generalization increasingly fails as more concept combinations are held out during training.This result concerns recombining known concepts when specific compositions are absent from training.

2. Related Work

Prior work shows that diffusion models often fail at multi-object compositional constraints, while existing compositional-generalization studies largely emphasize in-distribution or single-object settings. Controlled datasets provide useful supervision, but they do not fully capture discrete, unconstrained multi-object composition.

  • Multi-object failures in diffusion models: Benchmarks confirm that foundational text-to-image diffusion models remain unreliable when generated images must satisfy compositional constraints from text prompts.These models nonetheless achieve impressive visual fidelity and realistic image generation.
  • Compositional generalization in image diffusion models: Most generative diffusion studies evaluate in-distribution generalization rather than true compositional generalization.Only a few studies explicitly investigate compositional generation by carefully controlling the training data.
  • Compositional generalization in image diffusion models: Existing controlled compositional studies primarily use single-object settings with continuous inputs, limiting their relevance to discrete multi-object compositional challenges.Continuous inputs such as RGB values create a large space of possible concept combinations.
  • Compositional generalization in image diffusion models: Bradley (2025) studies object-length generalization, but explicit spatial conditioning makes the setting less comparable to unconstrained real-world generation.
  • Controlled compositional datasets: Controlled datasets range from single-object benchmarks to multi-object resources with annotations for locations, segmentation masks, language explanations, and depth.Examples include Shapes2D, 3D Shapes, CelebA, Kubric, Super-Clevr, and CLEVR-X.

3. MOSAIC: Diagnostic Dataset for Multi-Object Compositions

MOSAIC is a controlled dataset designed to isolate attribution, counting, and spatial-relation concepts in multi-object compositions. Its simulation-based construction varies selected factors while controlling other scene properties, enabling systematic difficulty and compositional-generalization experiments.

  • Dataset design: MOSAIC isolates three multi-object concepts: color attribution, counting, and spatial relations.It is presented as the first controlled dataset to separate these concepts.
  • Dataset design: The simulation-based pipeline varies one factor at a time while fixing or randomizing other properties and avoiding object occlusions.Blender renders photorealistic scenes with explicitly controlled parameters and exact ground truth.
  • Dataset subsets: Attribution evaluates object-color binding using sphere and cube identities, ten colors, and 100 possible sphere-cube color combinations.The combinations arise from 10 × 10 color pairings.
  • Dataset subsets: Spatial Relations uses ten angular relation classes around a randomly positioned brown reference sphere, while Counting varies object numerosity from one to ten.Counting becomes more spatially complex at higher counts because repeated objects must remain separated rather than collapse into fewer instances.
  • Experimental settings: MOSAIC varies difficulty through Complex and Grid settings and evaluates compositional generalization by combining multiple conditioning concepts.Complex adds objects for Attribution and Spatial Relations, Grid constrains Counting positions to predefined radial regions, and Composition adds Color to Counting and Spatial Relations.

4. Experimental Setup

The experiments separately test concept and compositional generalization by varying dataset size, concept balance, and held-out compositions. Models use controlled latent-diffusion training and task-specific classifiers to evaluate multi-object generation.

  • Experimental designs: The study separates concept generalization (RQ1) from compositional generalization (RQ2) and varies dataset size, concept balance, and unseen compositions independently.This design enables controlled analysis of both research questions.
  • Concept imbalance: For concept generalization, skewed and uniform regimes extend imbalance patterns across counts, angles, and colors for controlled task comparisons.Uniform assigns equal samples per category, while skewed distributions model unequal category frequencies.
  • Dataset sizes: Concept-generalization datasets contain 2k, 10k, 50k, or 100k samples, while compositional-generalization datasets contain 10k, 50k, or 100k samples.The compositional setting keeps seen-pair frequencies uniform.
  • Seen vs unseen compositions: Compositional generalization holds out 0, 1, 3, 5, or 8 diagonals while each individual concept remains observed during training.Remaining compositions are resampled to keep total dataset size comparable.
  • Training and evaluation: Training uses an approximately 90M-parameter latent diffusion architecture with a pretrained VAE, U-Net or DiT backbone, and attention-based conditioning.Task-specific discriminative classifiers evaluate generated scenes, using CNNs for Counting and ImageNet-initialized ResNet classifiers for Attribute Binding and Spatial Relations.

5. Concept Generalization

Concept generalization is driven more by scene complexity and dataset size than by concept imbalance. Counting is especially unstable in low-data regimes, while simpler spatial structure can partially alleviate its difficulty.

  • Concept imbalance: Attribution and Spatial Relations exceed 90% accuracy across dataset sizes, distributions, and both U-Net and DiT architectures, even under highly skewed training distributions.These tasks remain stable across all evaluated dataset sizes and data distributions.
  • Scene complexity: Scene complexity dominates concept-learning behavior, because Counting involves up to 10 objects while Attribution and Spatial Relations contain only two.Complex settings were introduced for Attribution and Spatial Relations to disentangle scene complexity from concept difficulty.
  • Low-data dynamics: Counting is uniquely difficult in low-data regimes: accuracy peaks early and subsequently deteriorates, despite smoothly decreasing training loss across dataset sizes and architectures.Attribution and Spatial Relations instead quickly saturate or converge without the same instability.
  • Spatial inductive bias: Reducing spatial complexity with a radial grid layout facilitates accurate counting, especially at moderate dataset sizes and in low-data regimes.The layout constrains each object to a designated canvas region with small positional jitter, providing a spatial inductive bias.

6. Compositional Generalization

Compositional generalization degrades as more concept combinations are held out, with limited benefit from scaling data. Spatial relations are especially difficult to learn compositionally, while attribution is comparatively robust and counting remains variable.

  • Experimental setup: The study controls the number of unseen compositions and dataset size to evaluate compositional generalization beyond densely covered single-object attribute combinations.The analysis focuses on the DiT architecture and uses a diagonal leave-out scheme.
  • Compositional generalization: Accuracy on unseen compositions improves with larger datasets but declines across all categories as more combinations are held out.This degradation occurs across the evaluated dataset sizes and categories.
  • Task differences: Attribution remains the most robust task on unseen compositions, whereas Spatial relations degrade more strongly and Counting shows lower, more variable performance.For seen compositions, Attribution and Spatial Relations remain consistently strong, while Counting improves noticeably with dataset size.
  • Error analysis: When half the compositions are held out, Attribution errors follow perceptual color similarity, while Counting predictions are typically one instance above or below the target.These near-label predictions indicate that the underlying concepts are at least partially learned.
  • Takeaway: Overall, increasing the number of held-out compositions exposes a fundamental limitation of current diffusion models, especially for Spatial Relations.The paper identifies compositional generalization failure as particularly severe for spatial relations.

7. Generalization to More Realistic Settings

The paper tests whether controlled MOSAIC findings persist under realistic scenes and pretrained-model fine-tuning. Counting remains unstable, while compositional generalization degrades as unseen object combinations increase, often collapsing to one object.

  • Realistic settings: The authors extend MOSAIC evaluations to text-prompt conditioning and richer object and background variations, including realistic scenes and object appearances.These additional experiments assess whether controlled-setting trends persist beyond MOSAIC’s diagnostic design.
  • Fine-tuning behavior: During LoRA fine-tuning of pretrained diffusion models, counting accuracy deteriorates on evaluation prompts even as spatial-relation accuracy improves.Training loss decreases for both tasks, but their evaluation behavior diverges.
  • Object co-occurrence: In a less controlled object-co-occurrence setting, accuracy on unseen object pairs degrades substantially as more compositions are held out, despite strong performance on seen pairs.The model often generates a single object or an incorrect secondary object.
  • Takeaway: Overall, controlled-setting behaviors persist under variations in appearance, viewpoint, and occlusion: counting remains unstable, and compositional generalization increasingly collapses to a single object.This summarizes the paper’s third takeaway for more diverse and realistic conditions.

8. Conclusion

Using MOSAIC, the study finds that data scale enables eventual generalization across spatial relations, counting, and attribution, while counting remains especially fragile with limited data and reduced scene complexity only partly helps.

  • Concept generalization: All three tasks eventually generalize when trained at sufficient scale under concept generalization.The tasks are Spatial Relations, Counting, and Attribution.
  • Concept generalization: Counting is particularly fragile in low-data regimes, and reducing scene complexity only partially mitigates this limitation.These findings concern the study’s analysis of data properties in multi-object generation.

Impact Statement

The work advances understanding of how data properties influence compositional generalization in conditional diffusion models. Its diagnostics and controlled benchmarks may support more reliable, robust, and interpretable multi-object generation.

  • Impact Statement: The study provides systematic diagnostics and controlled benchmarks for understanding how data properties affect compositional generalization in conditional diffusion models.These tools are intended to support development of more reliable generative models.
  • Impact Statement: The findings target multi-object generation, where robustness and interpretability remain limited, and may benefit applications requiring controllable image synthesis.The passage identifies downstream controllable image-synthesis applications as potential beneficiaries.

A. Appendix

The appendix supplements the main paper with detailed experimental setup, extended analyses, and qualitative examples covering MOSAIC, counting, and compositional generalization.

  • A.1–A.3: The appendix details experimental setup, including Figure 1(b), MOSAIC design choices, training, and evaluation.It also presents extended analyses of counting behavior and compositional generalization.
  • A.3: It concludes with qualitative training examples and generated images.

A.1. Experimental Setup · A.1.1. DETAILS FOR FIGURE 1 (B) IN THE MAIN PAPER

The setup quantifies how often counting and spatial-relation concepts occur in LAION-2B captions, then evaluates counting accuracy in SD3-medium using CompBench prompts. These analyses establish strong imbalance in relational supervision and a basis for studying data limitations in multi-object generation.

  • A.1. Experimental Setup: The study analyzes <count> + <object> phrase frequency in LAION-2B captions to assess how data limitations affect diffusion models.Captions with explicit number words are filtered, 5% are randomly sampled, and Qwen-8B is used to reduce rule-based extraction noise.
  • A.1. Experimental Setup: Counting evaluation uses SD3-medium with 830 uniformly generated CompBench prompts for each target count.The protocol uses UniDet to assess object presence and counting accuracy, but this evaluation reports only counting accuracy.
  • A.1.1. DETAILS FOR FIGURE 1 (B) IN THE MAIN PAPER: Spatial-relation frequency is measured by detecting <object> + <relation> + <object> patterns in captions containing at least one relational term.Relations are grouped into categories including right of, left of, above, below, next to, behind, and in front of.
  • A.1.1. DETAILS FOR FIGURE 1 (B) IN THE MAIN PAPER: Qwen-8B validates and refines the extracted spatial relations using prompts that count only explicitly stated object-position phrases.The prompts prohibit assumptions about implied positions and enumerate exact phrase variants for each relation category.
  • A.1.1. DETAILS FOR FIGURE 1 (B) IN THE MAIN PAPER: Figure A.1 shows substantial imbalance among spatial-relation terms in LAION-2B captions, extending the concept-imbalance analysis beyond counting.Some spatial relations occur far more frequently than others, indicating uneven relational supervision.
  • A.1.1. DETAILS FOR FIGURE 1 (B) IN THE MAIN PAPER: “in front of” appears more than 10 times as often as “left of” or “right of” in LAION-2B captions.This distribution highlights strong biases in relational supervision.

A.1.2. MOSAIC DESIGN DETAILS … A.3.3. GENERATED SAMPLES.

MOSAIC is a controlled, diagnostic framework for studying multi-object generation, with simplified scenes, systematic compositional holdouts, and standardized diffusion-model evaluation. Additional analyses show that low-data counting failures, overfitting, and large composition gaps remain difficult despite stable optimization, alternative encoders, or increased realism.

  • A.1.2. MOSAIC DESIGN DETAILS: MOSAIC uses photorealistic 512×512 3D assets in a controlled top-down environment without occlusions, limited object diversity, or other low-level visual confounds.The framework is intended diagnostically rather than as a realistic generative benchmark, while supporting future extensions with occlusions, distractors, varied viewpoints, and greater scene complexity.
  • A.1.2. MOSAIC DESIGN DETAILS: Compositional generalization is tested by removing k ∈{0, 1, 3, 5, 8} diagonals from concept-pair matrices and resampling remaining compositions to keep dataset sizes comparable.Earlier removed diagonals increase difficulty while all individual concepts remain observed.
  • A.1.3. TRAINING DETAILS.: Models are trained with AdamW on four A100 GPUs, using 128×128 images, 20k steps unless noted, and a default learning rate of 0.0001 selected for the 90M baseline.The 0.0001 learning rate yields 0.962 validation accuracy for the 90M model on Counting (100k, Uniform).
  • A.1.4. EVALUATION DETAILS: Evaluation uses pretrained classifiers for 20 count classes, 10 spatial-relation classes, and 100 attribution color-pair classes, with Accuracy and Memorization rate as the main metrics.Across settings, 50 samples per condition are generated with DDIM sampling without classifier-free guidance.
  • A.2. Additional Analysis: Counting accuracy drops sharply at small dataset sizes regardless of skewness, recovers with larger datasets, and follows the same trend across model capacities.At 10k samples, accuracy progressively collapses toward under-counting, whereas only sufficiently large datasets stabilize performance across counts.
  • A.2.1. COUNTING BEHAVIOR ANALYSIS: At 2k samples, all three tasks exhibit near-100% memorization, which gradually decreases with dataset scale as models enter an intermediate regime where memorization is no longer feasible.Lower-count classes are memorized first, while higher counts remain harder to memorize.
  • A.2.2. COMPOSITIONAL GENERALIZATION ANALYSIS: Compositional accuracy drops sharply when more than half of compositions are held out, with Counting and Spatial Relations collapsing while Attribution remains comparatively robust; grid layouts provide moderate gains but color accuracy declines after roughly half the diagonals are held out.Extended training plateaus for both seen and unseen compositions, and a disentangled frozen encoder provides only marginal improvements, so encoder quality alone does not recover compositional generalization.
Loading 2605.00273v2…