Source-linked AI summary
Make It Count: Text-to-Image Generation with an Accurate Number of Objects
Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, Gal Chechik
TL;DR
Text-to-image diffusion models struggle to generate the requested number of objects because they must preserve separate instance identities while enforcing global spatial constraints. CountGen identifies objectness features, counts instances during denoising, corrects layouts with ReLayout, and guides generation; it improves count accuracy across two benchmarks, from 29% to 48% and from 26% to 54% relative to SDXL. The approach is evaluated on scenes with up to 10 instances and a single object per prompt.
Problem
Text-to-image diffusion models often fail to generate the correct number of requested objects, a task requiring objectness and global spatial control during image generation.
Method
CountGen identifies objectness and instance-identity features, counts objects during denoising, trains ReLayout to add or remove instances while preserving layout structure, and guides generation from corrected layouts.
Results
48% human-evaluated accuracy on T2I-CompBench versus 29% for SDXL, and 54% CoCoCount accuracy versus 26% for SDXL, with improvements over all tested baselines.
Takeaways & Limitations
CountGen shows that diffusion-model representations can support count-aware generation through intermediate instance detection and layout correction.
Takeaways & Limitations
The experiments focus on scenes with up to 10 instances and a single object per prompt, and optimization can sometimes create duplicate instances or plainer backgrounds.
Abstract
from arXiv · showhide
Despite the unprecedented success of text-to-image diffusion models, controlling the number of depicted objects using text is surprisingly hard. This is important for various applications from technical documents, to children's books to illustrating cooking recipes. Generating object-correct counts is fundamentally challenging because the generative model needs to keep a sense of separate identity for every instance of the object, even if several objects look identical or overlap, and then carry out a global computation implicitly during generation. It is still unknown if such representations exist. To address count-correct generation, we first identify features within the diffusion model that can carry the object identity information. We then use them to separate and count instances of objects during the denoising process and detect over-generation and under-generation. We fix the latter by training a model that predicts both the shape and location of a missing object, based on the layout of existing ones, and show how it can be used to guide denoising with correct object count. Our approach, CountGen, does not depend on external source to determine object layout, but rather uses the prior from the diffusion model itself, creating prompt-dependent and seed-dependent layouts. Evaluated on two benchmark datasets, we find that CountGen strongly outperforms the count-accuracy of existing baselines.
1 Introduction
Text-to-image diffusion models often fail to generate the requested number of objects because counting requires object identity and spatial control during generation. CountGen detects instances during denoising, corrects layouts, and substantially improves count accuracy over existing methods.
- Motivation: Manual layouts and large vision-language models provide imperfect alternatives because they lack prompt-dependent layouts or fail to exploit the diffusion model’s visual prior.Manual layouts require designing spatial organization for each count, while vision-language-model proposals perform poorly on counting.
- Problem: Counting remains difficult because models must represent each object as a separate entity while controlling global spatial layout during image generation.These failures are especially obvious when prompts specify a particular number, such as three bears.
- Method: CountGen identifies self-attention features encoding objectness and instance identity, then counts object instances during intermediate denoising steps.Cross-attention maps localize the features, which are clustered into object-instance segmentations before generation finishes.
- Method: ReLayout adds a missing object by predicting a natural location from the layout of existing objects, enabling count correction without externally specified layouts.The model is trained on image-pair samples generated by the diffusion model itself, and the corrected layouts guide test-time denoising optimization.
- Results: 48% human-evaluated accuracy on T2I-CompBench and 54% detector-evaluated accuracy on CoCoCount, versus 29% and 26% for SDXL, respectively.CountGen also improves over other baselines, including DALL-E 3, on both benchmarks.
- Contribution: The paper reports state-of-the-art count-accurate generation through objectness features, inference-time guidance, and automatic layout modification.These contributions jointly form CountGen.
2 Related Work
Prior work addresses incorrect object counts through counting guidance, text-to-layout decomposition, predefined layouts, and attention-based losses. These approaches either add auxiliary models, depend on externally proposed layouts, or target related failures such as object neglect and attribute binding.
- Counting accuracy: Classifier-guidance methods count instances at each diffusion step and adjust denoising, but require an additional U-Net during every step.This work is presented as a prior approach to improving object-count accuracy.
- Layout-based generation: A major line of work separates generation into text-to-layout and layout-to-image stages, with layouts specifying spatial locations for object instances [3] [25].LayoutGPT injects visual commonsense into prompts, while another approach decomposes complex prompts before layout generation.
- Layout-based generation: Predefined layouts with the exact number of subjects help ensure that generated images contain the requested count.This strategy treats spatial layout as an explicit conditioning signal for layout-to-image generation.
- Attention-based control: Attention-based losses address object neglect by ensuring prompted objects appear in cross-attention maps and address attribute association by binding subject and attribute maps [7] [23].These methods target prompt-object correspondence and attribute binding rather than the full counting problem.
3 Our Approach: CountGen
CountGen extracts instance-level information from SDXL’s early denoising representations, corrects mismatched object counts while preserving scene composition, and guides generation with the resulting layout.
- 3.1 Discovering Object-Instance Layout during Early Generation: CountGen builds a prompt- and seed-dependent layout by extracting object instances from SDXL’s naturally emerging generation rather than relying on externally specified layouts.It combines cross-attention maps with self-attention features to localize and cluster object instances during early denoising.
- 3.1 Discovering Object-Instance Layout during Early Generation: At timestep t = 500, selected SDXL self-attention features separate instances of the same object, providing the representation used for instance-level localization.The analysis compares features across layers and timesteps; the selected layer shows robust separation through distinct instance features.
- 3.1 Discovering Object-Instance Layout during Early Generation: Cross-attention foreground masks and clustered self-attention features produce one mask per generated object, which is grouped into an initial object layout.The method aggregates object-token cross-attention, thresholds it to identify foreground pixels, and clusters corresponding self-attention patches.
- 3.2 ReLayout: Correcting the Number of Objects in the Mask: ReLayout corrects over-generation by removing smallest instances and handles under-generation by inserting instances while preserving the original scene structure.When the count is already correct, ReLayout keeps the initial layout unchanged.
- 3.2.1 Handling Under-generation: For under-generation, a U-Net iteratively predicts a layout mask with one additional instance from an existing layout.Its training data uses image pairs generated with fixed seeds and otherwise matching prompts that differ only in object count, preserving scene composition while adding one object.
4 Experiments
CountGen is evaluated against six baseline methods on two counting-focused datasets using human and automatic count-accuracy measures, alongside an image-quality comparison.
- Compared methods: CountGen is compared with SDXL, Repeated Object SDXL, Reason Out Your Layout, DALL-E 3, Random masks + BoundedAttn, Counting Guidance, and Bounded Attention.
- Datasets: The evaluation uses T2I-Compbench-Count prompts with 2–10 objects and the automatically evaluable CoCoCount dataset.
- Count accuracy evaluation: Count accuracy is assessed by whether generated instances match the number requested in the prompt, using human and standardized automatic evaluation.
- Human evaluation: Human raters check object presence, instance well-formedness, and the number of instances before assigning count accuracy.
- Automatic evaluation: Automatic evaluation counts target-object bounding boxes detected by YOLOv9 using its default settings.
- Image quality evaluation: Image quality is evaluated by asking raters to compare whether CountGen or SDXL produces the more natural and well-formed image.
5 Results
CountGen improves count accuracy across the evaluated benchmarks and qualitative examples, while DALL-E 3 slightly leads for two- and three-object cases and CountGen shows little reported quality loss.
- Quantitative results: CountGen significantly improves over competing baselines on both CoCoCount and T2I-Compbench-Count, with DALL-E 3 slightly outperforming it for two and three instances.
- Image quality: Only 23 of 200 image comparisons had a majority preferring SDXL over CountGen, indicating no significant reported loss of image quality.
- Qualitative results: CountGen consistently generates the correct number of object instances in qualitative comparisons where other methods struggle in some or all examples.
6 Ablation Study
Ablations show that both CountGen-Layout and CountGen-Image improve accuracy, while layout loss and self-attention masking jointly maintain adherence to the required object layout.
- Component contributions: Together, CountGen-Layout and CountGen-Image improve human-rated accuracy by 21 points over Random Masks + Bounded Attention.
- Component contributions: CountGen-Layout improves accuracy by 14% from 30 to 44, while CountGen-Image contributes an additional 12%.
- Layout-guided generation ablation: Removing the layout loss causes objects to scatter beyond the required layout, whereas removing self-attention masking allows unwanted background instances.
- Count accuracy: CountGen outperforms all methods for more than three objects and matches DALL-E 3 for two and three objects in human-rated accuracy.
- Layout adherence: Without self-attention masking, precision worsens; without layout loss, recall and IOU fall, while both components together provide balanced mask adherence.
7 Limitations
The experiments are limited to scenes with up to 10 instances and one object category per prompt, and optimization can occasionally duplicate objects within a single intended region.
- Failure cases: CountGen occasionally generates multiple object instances in an area where the layout intends only one.
- Scope: The experiments focus on scenes with at most 10 instances and a single object per prompt, limiting the evaluated scope.
8 Conclusions
CountGen addresses the difficulty of generating the requested number of objects by combining objectness-based counting, layout correction, and layout-guided denoising. On the stated task, it nearly doubles SDXL counting accuracy, but adds substantial generation time.
- 8 Conclusions: Accurate count generation remains difficult because models must preserve objectness, global spatial constraints, and natural image formation simultaneously.Existing text-to-image diffusion models perform especially poorly beyond three requested objects.
- 8 Conclusions: 54% counting accuracy with CountGen versus 26% for standard SDXL, nearly doubling performance.The comparison is reported for CountGen applied to SDXL.
- 8 Conclusions: CountGen identifies objectness in self-attention, corrects layouts by adding or removing instances, and guides generation from the corrected layout.The three stages are presented as the approach to accurate object-count generation.
- 8 Conclusions: CountGen requires about 36 seconds per image on one A100 80GB, compared with about 55 seconds for Bounded-Attention.The reported timing is obtained by iterating over CoCoCount; Bounded-Attention also requires bounding boxes as input.
B.2 Compared Methods
The comparison includes diffusion, prompt-editing, layout-generation, and counting-guidance baselines, while ReLayout learns to add an instance without disrupting existing object structure. CountGen outperforms all methods in the supplied human- and YOLOv9-based accuracy figures.
- B.2 Compared Methods: Human evaluation shows CountGen ahead for more than three objects and comparable to DALL-E 3 for two and three objects.The evaluation covers 200 images.
- B.2 Compared Methods: CountGen outperforms all methods in YOLOv9 accuracy across the number of generated objects.The figure evaluates accuracy over 200 images.
- B.2 Compared Methods: ReLayout predicts existing-mask changes and the placement of one additional object while preserving spatial and shape consistency.The added object is represented by the unmatched mask after Hungarian matching between layouts.
- B.2 Compared Methods: The ReLayout objective uses Dice loss for mask overlap and intersection loss to reduce overlap between distinct predicted object masks.The supplied passages describe both losses as training objectives for the output masks.
C.3 Datasets
CoCoCount contains 200 prompts spanning 20 MSCOCO object classes, six requested-count categories, and optional scene contexts. Automatic evaluation uses the large YOLOv9e detector implementation from Ultralytics.
- C.3 Datasets: CoCoCount contains 200 prompts built from 20 randomly selected MSCOCO classes and counts of 2, 3, 4, 5, 7, or 10.The two- and three-object categories contain 34 samples each; the remaining categories contain 33.
- C.3 Datasets: Prompts follow “a photo of {number} {object}” and add “on the grass,” “on the road,” or “on the ground” for 50% of prompts.Scene variations are incorporated randomly.
- C.3 Datasets: The object vocabulary includes animals, vehicles, accessories, food, and everyday items such as cars, cats, cups, apples, and clocks.The complete supplied list contains 20 object classes.
- C.3 Datasets: Automatic evaluation uses Ultralytics YOLOv9e, the large YOLOv9 implementation.This detector is used for the automatic evaluation procedure.
D.1 Human Evaluation
Human evaluation uses screened Amazon Mechanical Turk raters, majority decisions from three raters per example, and separate tasks for counting, well-formedness, and fidelity. The supplied passages identify the task materials and confusion-matrix visualization but do not report further numerical outcomes.
- D.1 Human Evaluation: Each example was judged by three Mechanical Turk raters, with the majority selection used as the result.Raters needed at least 5,000 approved HITs and an approval rate above 98%.
- D.1 Human Evaluation: The reported compensation was $15 per hour.The evaluation used Amazon Mechanical Turk.
- D.1 Human Evaluation: The evaluation includes tasks for counting objects and assessing well-formedness, prompt matching, and naturalness.The supplied figures show instructions and example tasks for these judgments.
- D.1 Human Evaluation: The count-accuracy experiment includes a human-evaluation confusion matrix for CountGen.The supplied passage identifies this as Figure 17 without reporting its cell values.