Source-linked AI summary
NumBench: Diagnosing Counting Failures in Text-to-Image Models
Sandeep Wadhwa, Mayank Vatsa, Richa Singh, Parrva Chirag Shah, Prakhar Galriya
TL;DR
Existing benchmarks provide limited, weakly controlled evidence for diagnosing counting failures in text-to-image models. NumBench combines a large factorial benchmark, a spatial-collision process model, and calibrated detector-based evaluation, finding that requested count dominates difficulty and performance collapses at high counts.
Problem
Existing counting evaluations lack broad count ranges, category diversity, controlled spatial variation, and reliable detector-based measurement for diagnosing text-to-image counting failures.
Method
NumBench combines 640,000 controlled prompts across 1,600 categories and counts 1–100 with a spatial-collision process model and calibrated detector-ensemble metric.
Results
Requested count is the dominant difficulty, with every evaluated method declining as counts rise and remaining weak above 50 objects.
Takeaways & Limitations
Grid guidance performs best among guided layouts, while specialized methods improve low-count results but do not solve high-count generation.
Takeaways & Limitations
The detector-mediated metric and human reference become less reliable above 50 objects, so that regime is treated as a stress test rather than an equally precise count estimate.
Abstract
from arXiv · showhide
Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We introduce \textbf{NumBench}, a benchmark of 640{,}000 prompts spanning 1{,}600 categories and counts from 1 to 100. Its factorial design varies object composition, spatial guidance, and appearance conditions while balancing counts and category exposure. We also develop a process model in which requested instances compete for a finite set of resolvable image regions. The model predicts a near-quadratic collision deficit at low occupancy and shows how coordinated placement reduces it. For scalable evaluation, we propose the Confidence-Weighted Numeric Precision Score (\cwnps), which aggregates three calibrated detectors and discounts uncertain proposals. Across five commercial systems, two open models, and two specialized counting methods, performance declines sharply with requested count; all evaluated methods are weak above 50 objects. Count range has the largest measured effect, followed by layout and composition. Grid guidance is strongest among guided layouts, consistent with the coordination prediction, although the analysis does not establish collision as the sole cause. A 14{,}400-image human study supports automated evaluation through count 50, while results on 243 natural-language prompts show transfer beyond NumBench templates.
Introduction
NumBench addresses persistent counting failures in text-to-image systems with a large, controlled benchmark, a spatial-collision process model, and a confidence-weighted evaluation metric. Results identify requested count as the dominant difficulty, with grid layouts helping while counts above 50 remain a stress-test regime.
- Introduction: Counting failures include missing or merged objects, category confusion, and repeated default patterns, and existing benchmarks do not isolate their causes.Prior general benchmarks contain small counting subsets, while counting-specific sets have limited count range, category diversity, or controlled spatial variation (Saharia et al. 2022; Ghosh, Hajishirzi, and Schmidt 2023; Huang et al. 2023; Bakr et al. 2023; Hu et al. 2023; Kajić et al. 2024; Binyamin et al. 2025; Sun et al. 2026; Zafar et al. 2026).
- Introduction: The confidence-weighted cw-NPS metric addresses detector misses that can otherwise be mistaken for generator errors, especially in crowded scenes.It is proposed to aggregate calibrated detectors while discounting uncertain proposals.
- Introduction: NumBench contributes 640,000 prompts spanning 1,600 categories and counts from 1–100 under controlled composition, layout, and appearance conditions.The design crosses single- versus double-category composition, free-form versus guided layout, and clean versus modified appearance instructions.
- Introduction: Its spatial-collision process model predicts rapidly growing undercount from limited spatial capacity and reduced loss under coordinated layouts.The model explains why requested instances compete for a finite set of resolvable image regions.
- Introduction: Requested count is the dominant difficulty, followed by layout, composition, and appearance; grids perform best among guided layouts, while counts above 50 remain weakly validated stress tests.Human agreement and detector–human correlation weaken above count 50, so this region is treated as a stress test rather than a fully validated estimate.
Related Work
Prior benchmarks assess counting within broader text-to-image compositionality evaluations, while newer resources focus specifically on counting. These focused resources include GeckoNum, CoCoCount, QUANT-Bench with QUOTA, and an FSC-147-based protocol.
- Benchmarks: Broader compositionality benchmarks evaluate counting as one component, including DrawBench (Saharia et al. 2022), PaintSkills (Cho, Zala, and Bansal 2023), GenEval (Ghosh, Hajishirzi, and Schmidt 2023), T2I-CompBench, HRS-Bench (Bakr et al. 2023), and TIFA (Hu et al. 2023).
- Benchmarks: Focused counting resources include GeckoNum (Kajić et al. 2024), CoCoCount (Binyamin et al. 2025), QUANT-Bench introduced with QUOTA, and the FSC-147-based protocol of Zafar et al..
A Process Model for Spatial Collisions
The process model treats counting errors as collisions among requested instances competing for a finite set of resolvable image regions. It predicts a near-quadratic low-occupancy deficit, while coordinated placement reduces collisions and motivates—but does not causally establish—regular-layout advantages.
- Collision model: At low occupancy, independent assignment of n instances to M resolvable regions produces an expected visible deficit that grows approximately quadratically with n/M.The mechanism is pairwise competition for the same region, which can merge requested instances into one visible instance.
- Scope and limitations: The model explains one collision mechanism only, not overcounting, category substitution, or learned preferences for common scene sizes.Increasing canvas resolution helps only if it increases usable regions at rendered object scale; post-generation upsampling cannot recover merged instances.
- Layout implications: The model predicts that regular layouts should degrade more slowly than weakly coordinated layouts when guidance increases λ, effective capacity M, or both.The experiments use layout ordering as a diagnostic consistency check rather than causal identification.
- Coordination: Coordination reduces the leading collision deficit by replacing 1/M with q/M, where q = 1 −λ; when λ = 1, every instance occupies an unused region and R = n.Here λ is the probability of selecting an unused region.
- Scope and limitations: Counting improvements from layout are observationally distinct only with directional count errors and controlled object scale, conditions NumBench enables for future tests.A layout method may improve counting by increasing M, λ, or both.
NumBench
NumBench is a balanced counting benchmark built from 160,000 core prompts across 1,600 categories and counts from 1 to 100. It expands these prompts across controlled layout and appearance conditions while keeping quality degradation distinct from instance-separation difficulty.
- Core benchmark: NumBench’s 160,000 core prompts cover 1,600 curated categories using single- and double-category compositions.The vocabulary removes synonyms, plural variants, and formatting duplicates, and draws from seven established datasets.
- Core benchmark: Category and count assignments are marginally balanced: single-category prompts contain 50 instances per category and 800 instances per integer, while double-category prompts double both totals.Counts are independently assigned for the two categories rather than forming a complete category–count crossing.
- Controlled expansions: The benchmark expands along layout and appearance axes, comparing free-form prompts with balanced grid, scene, and random guided forms.Each core prompt has a free-form reference but only one guided counterpart, so factor summaries treat the four layout levels separately rather than pooling guided prompts.
- Controlled expansions: Appearance conditions separate image-quality degradation from changes in the visible evidence used by detectors or people to infer cardinality.This distinction motivates treating the two appearance groups as semantically different analytical factors.
Confidence-Weighted Numeric Precision
Confidence-Weighted Numeric Precision combines calibrated proposals from three detectors with reference-conditioned thresholds to reduce single-detector and hard-threshold weaknesses. The score is fixed before evaluation, uses requested count, and is therefore an evaluation metric rather than an independent count estimator.
- Metric construction: Confidence-Weighted Numeric Precision aggregates OW-DETR, Grounding DINO, and OWL-ViT proposals after category-specific non-maximum suppression, reducing dependence on any single detector.Each detector is queried through its native category interface before independent class-aware suppression and aggregation.
- Calibration: Temperature scaling calibrates detector confidences using a disjoint 5,000-image set, with fitted temperatures 1.15, 1.42, and 0.93 for OW-DETR, Grounding DINO, and OWL-ViT.The values correspond to OW-DETR, Grounding DINO, and OWL-ViT, respectively.
- Thresholding: The score lies in [0, 1], uses a pre-specified reference-conditioned threshold schedule, and begins changing above count 3 before reaching .3 at count 10.Calibration and thresholding are fixed before evaluating all nine systems, and the schedule is not claimed to be optimal.
- Scope and limitations: Because its threshold schedule depends on requested count, cw-NPS is an evaluation score rather than an independent count estimator.The schedule favors proposal recall in crowded scenes, while paired b-NPS results test dependence on confidence weighting alone.
Experiments
Across commercial, open, and specialized systems, counting performance declines sharply as requested counts increase, with all evaluated methods weak above 50 objects. Count range is the strongest measured factor, while composition, layout, appearance, detector choice, and validation regime materially affect interpretation.
- Composition effects: Composition changes rankings: CountGen leads single-category ranges, whereas Bounded Attention leads double-category ranges, reaching .641 versus .610 at counts 1–25.Double-category prompts are not uniformly harder for every system and count range.
- Factor analysis: Appearance condition has a mean best–worst difference of .45, with overlap, merging, cropping, and partial objects directly altering visible counting evidence.Other appearance changes primarily affect visual quality rather than counting evidence.
- Overall benchmark results: Every method degrades with count: CountGen falls from .692 to .129 and Bounded Attention from .623 to .110 across single-category ranges, while commercial systems reach at most .16 at counts 76–100.Table 4 reports degradation under both basic and confidence-weighted precision; specialized methods help at low and moderate counts but do not solve high-count generation.
- Factor analysis: Count range has the largest measured factor effect, and grid guidance has the highest guided-layout score at .45 versus .31 for scene and .32 for random.The factor contrasts are descriptive associations rather than causal coefficients, and the layout comparison lacks a matched free-form aggregate.
- Validation and transfer: Human agreement is strongest through count 50, while above-50 results are stress tests; on 243 natural-language prompts, rankings remain broadly consistent and no system exceeds .67 cw-NPS.Replacing the detector ensemble with any single detector shifts aggregate cw-NPS by at least .10, so detector choice materially affects measurement.
Limitations and Broader Impact
NumBench’s evaluation remains detector-mediated and dataset-bounded: detector biases, threshold choices, source-dataset scope, and changing commercial snapshots limit interpretation. Transfer results show that perceptual quality does not predict numeric precision, while the benchmark’s ethical and release practices support reproducible evaluation without adding image-generation capability.
- Measurement validity: cw-NPS remains detector-mediated, so shared detector biases and threshold choices can affect measured counting errors.The count-conditioned threshold favors recall in crowded scenes, whereas a fixed threshold can exaggerate undercount as confidence falls with object size; the paper therefore reports absolute precision alongside b-NPS.
- Transfer: No evaluated system exceeds .67 cw-NPS on 243 CoCoCount+ prompts, and perceptual quality does not predict numeric precision.The transfer examples expose exact-count, color-binding, missing-object, and overcount failures even at low counts.
- Broader impact: NumBench separates three failure families that require different evidence and reporting, so a single total-count score is insufficient.The benchmark excludes offensive labels and sensitive personal attributes, redistributes commercial outputs only when permitted, and introduces no new image-generation capability.
- Scope and reproducibility: NumBench’s categories inherit the visual scope of source datasets, while unversioned commercial snapshots collected between January 2025 and July 2026 may change after service updates.These boundaries motivate releasing prompts, timestamps, detector proposals, and open-model outputs for evaluation under the same protocol.
Conclusion
NumBench frames T2I counting as a controlled diagnostic problem across counts 1–100, 1,600 categories, and 640,000 prompts. Its collision model and cw-NPS enable scalable evaluation, revealing that requested count dominates measured factors and that high-count numeric precision remains weak.
- NumBench covers counts 1–100, 1,600 categories, and 640,000 prompts, turning T2I counting into a controlled diagnostic problem.
- A spatial-collision model explains how limited coordination can produce rapidly growing visible counting deficits, while cw-NPS supports evaluation at scale.
- Across nine systems, requested count dominates all measured factors, while layout and composition also affect counting performance.
- No evaluated method retains strong numeric precision at high requested counts.