Source-linked AI summary
Teaching CLIP to Count to Ten
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, Tali Dekel
TL;DR
Existing VLMs have limited compositional understanding of object counts, despite their broad usefulness in vision and multimodal tasks. The paper trains pretrained VLMs with accurate counting data and count-based hard negatives, then evaluates them on counting and downstream applications. The resulting models improve counting and count-specific retrieval and generation while retaining common zero-shot performance, with generalization beyond ten remaining unclear.
Problem
Existing VLMs struggle to associate visible object counts with numbers in captions, partly because accurately counted captions are rare and standard objectives do not make counting decisive.
Method
The method filters accurate image-caption count pairs and finetunes a pretrained VLM with a counting-contrastive loss that separates true-count captions from counterfactual incorrect counts.
Results
The method significantly improves zero-shot count classification for CLIP and BASIC while maintaining performance on standard zero-shot tasks and improving count fidelity in retrieval and generation.
Takeaways & Limitations
Counting-aware VLM representations support more reliable retrieval and text-conditioned generation when prompts specify object counts.
Takeaways & Limitations
The method is limited by scarce accurately counted multi-object training data and currently teaches CLIP to count only through ten, with higher-number generalization unclear.
Abstract
from arXiv · showhide
Large vision-language models (VLMs), such as CLIP, learn rich joint image-text representations, facilitating advances in numerous downstream tasks, including zero-shot classification and text-to-image generation. Nevertheless, existing VLMs exhibit a prominent well-documented limitation - they fail to encapsulate compositional concepts such as counting. We introduce a simple yet effective method to improve the quantitative understanding of VLMs, while maintaining their overall performance on common benchmarks. Specifically, we propose a new counting-contrastive loss used to finetune a pre-trained VLM in tandem with its original objective. Our counting loss is deployed over automatically-created counterfactual examples, each consisting of an image and a caption containing an incorrect object count. For example, an image depicting three dogs is paired with the caption "Six dogs playing in the yard". Our loss encourages discrimination between the correct caption and its counterfactual variant which serves as a hard negative example. To the best of our knowledge, this work is the first to extend CLIP's capabilities to object counting. Furthermore, we introduce "CountBench" - a new image-text counting benchmark for evaluating a model's understanding of object counting. We demonstrate a significant improvement over state-of-the-art baseline models on this task. Finally, we leverage our count-aware CLIP model for image retrieval and text-conditioned image generation, demonstrating that our model can produce specific counts of objects more reliably than existing ones.
1. Introduction
Existing VLMs struggle with compositional counting, so the paper introduces a counting-focused training framework and benchmark. The method improves counting while preserving common zero-shot performance and supports more reliable count-specific retrieval and generation.
- Motivation: CLIP and BASIC often fail to represent object counts, limiting zero-shot retrieval and count-specific image generation.These models tend to prioritize other image-text features over compositional quantities.
- Method: The framework creates accurate counting examples and trains a pretrained VLM to distinguish true-count captions from counterfactual captions with incorrect numbers.The counterfactual changes the object count while preserving the rest of the caption, making counting critical for discrimination.
- Results: The method significantly improves zero-shot count classification for CLIP and BASIC while maintaining accuracy on common zero-shot benchmarks.The evaluation covers both counting performance and retention of the VLMs’ original knowledge.
- Evaluation: CountBench is a carefully filtered and validated benchmark for evaluating vision-language counting.The benchmark is introduced specifically to measure VLM understanding of object counts.
- Downstream applications: Counting-aware VLMs produce more reliable object counts in image retrieval and text-conditioned generation.The paper reports improved count fidelity in downstream retrieval and generated images.
2. Related work
Prior work established contrastive VLMs as broadly useful but showed that they only partially capture compositional meaning, including object counts. This paper addresses counting through a counting-specific contrastive framework rather than a VQA-only architecture.
- Contrastive vision-language models: Contrastive VLMs learn joint image-text representations from large paired datasets and support zero-shot or finetuned vision and multimodal applications.CLIP uses a contrastive objective that brings matching image-text pairs together and separates non-matching pairs.
- Compositionality and counting: Large VLMs such as CLIP and BASIC have been shown to capture compositional meaning only partially, including relations between objects and their numbers.Prior analysis found that CLIP often attends mainly to nouns while ignoring adjectives, numbers, and prepositions.
- Counting in VLMs: Earlier counting approaches primarily addressed visual question answering with specialized architectures, whereas this work targets general-purpose contrastive VLMs.The paper distinguishes its setting from counting-specific VQA formulations and architectures.
- Text-conditioned generation: CLIP is also widely used as a representation component in text-conditioned image generation.Prior systems use CLIP representations for latent optimization, classifier guidance, or image editing.
3. Method
The method builds a verified, balanced counting dataset and fine-tunes a pretrained VLM with a count-specific contrastive loss alongside CLIP’s original objective.
- 3. Method: Counterfactual captions are created by replacing the true object count with a randomly selected different number.For example, “five dogs” can become “eight dogs.”
- 3.2. Teaching CLIP to count: The counting loss makes the image more similar to its true-count caption and less similar to the counterfactual caption, which differs only in the spelled number.This forces the model to learn the relationship between the specified number and the referenced objects.
- 3.1. Creating an image-text counting train set: The pipeline filters captions containing spelled numbers, removes numbers referring to non-count attributes, and verifies that counted objects are visible and detectable.It retains examples when the captioned number matches the maximally detected object count.
- 3.1. Creating an image-text counting train set: The filtered data are balanced separately for counts two–six and seven–ten because examples become much rarer as the count increases.Approximately 37K samples are used for each count from two through six; higher counts use all retained samples, ranging from about 7K for seven to 1.5K for ten.
- 3.2. Teaching CLIP to count: The objective combines CLIP’s regular contrastive loss with the counting loss, weighted by the hyperparameter λ.The regular loss is applied to all samples, whereas the counting loss is applied only to the filtered numbered training set C.
4. CountBench
CountBench is a held-out, manually verified benchmark for object counting, built from LAION-400M examples spanning two to ten instances. Its higher-count examples are increasingly biased toward simplistic 2D object collections rather than real-world scenes.
- CountBench contains 540 images with captions reflecting two to ten instances of a particular object.The benchmark is automatically curated from LAION-400M and manually verified.
- CountBench is used only for testing, with images filtered to avoid overlap with the counting training set C.This separates benchmark evaluation from training data.
- The benchmark is automatically balanced and manually verified so each caption’s spelled number matches the clearly visible object count.Filtering initially produced over 158K images for “two” but only around 100 for “ten”; numbers were balanced to 100–200 samples each before verification.
- Higher counts contain a larger proportion of simplistic 2D object collections instead of objects in real-world scenes.The same bias also appears in web-scraped counting training data generally.
5. Experiments
Experiments show that counting-aware finetuning substantially improves zero-shot object-count prediction while preserving common zero-shot classification performance. Qualitative retrieval and relevancy analyses further indicate improved sensitivity to requested counts and count-relevant image regions, within a scope limited by sparse high-count training data.
- 5. Experiments: The evaluation combines CountBench zero-shot count classification with qualitative retrieval, relevancy-map analysis, and standard zero-shot classification benchmarks.
- 5.1. Zero-shot counting accuracy: 2–3ˆ higher counting accuracy and more than 3ˆ lower mean deviation distinguish counting-aware CLIP and BASIC from their corresponding baselines.
- 5.1. Zero-shot counting accuracy: The counting loss and restrictive filtering pipeline are both important, since removing either component lowers CountBench counting accuracy.Configuration C omits the counting loss, whereas configuration D uses counting loss with naively filtered data.
- 5.1. Zero-shot counting accuracy: Across numbers, the improved CLIP model outperforms the public baseline, although accuracy drops for some higher counts because those counts are underrepresented in training data.
- 5.1. Zero-shot counting accuracy: The counting-aware models maintain similar overall accuracy on common non-counting zero-shot classification benchmarks.
- 5.2. Count-based image retrieval: For requested counts above three, baseline retrieval often shows arbitrary or repeated object counts, whereas the counting-aware model depicts accurate counts in most cases.The retrieval study uses qualitative results because the dataset lacks ground-truth object-count labels.
- 5.2. Count-based image retrieval: Relevancy maps assign greater importance to spelled numbers and focus the improved model on image areas relevant for counting.
- 5.5. Limitations: The method is constrained by scarce multi-instance images with captions giving accurate counts, especially for larger numbers and realistic scenes.
6. Conclusions and future work
The work enhances CLIP with counting abilities through counting-focused hard negatives and an additional loss, and introduces CountBench for evaluation. The resulting representations improve retrieval and generation of specific object counts, while the method remains limited to counting in this study.
- Downstream applications: 1.5–2× improvement is reported for Imagen's ability to generate specific object counts with counting-aware CLIP.The comparison is against Imagen trained with the official CLIP.
- Contributions: The method uses a clean counting subset, counting-focused hard-negative augmentation, and an additional loss to separate false captions from images.The training data are filtered from internet-collected datasets before counterfactual augmentation.
- Downstream applications: Counting-aware CLIP produces more accurate count-based image retrieval results than the compared model.Figure 8 orders retrieved images by similarity and marks matching results green and nonmatching results red.
- Downstream applications: Imagen can generate images that follow specified object amounts more accurately when conditioned on counting-aware CLIP representations.The reported generation examples use captions specifying object counts.
- Limitations: The method struggles with captions requiring prior knowledge, grid orientations associated with other counts, or quantity words such as “couple”.These cases are presented as exemplary failures.
- Contributions: CountBench contains in-the-wild images and captions that specify the number of particular objects.The benchmark is intended to support research on image-text counting.
- Limitations: The study focuses on counting because it is comparatively unambiguous to define and evaluate; extending the method to spatial positioning and verb roles remains future work.The authors state that the method is not specific to counting, but do not evaluate those other concepts here.
- Societal impact: Improved numerical representations may support downstream synthesis and editing applications, but malicious parties could use them to create misleading imagery.The authors distinguish this contribution from proposing a new image-generation method itself.
A. Image generation experiments
This section provides further details about the text-conditioned image-generation experiments described in Section 5.3.
- Scope: The appendix details the text-conditioned image-generation experiments introduced in Section 5.3.No additional experimental result is stated in this passage.
- Scope: The experiments concern text-conditioned image generation.The passage identifies the experiment type but does not specify settings or outcomes.
- Scope: The detailed experimental description is presented outside the main-text discussion referenced by this section.The passage frames the section as supplementary detail.
A.1. Experiment settings
The experiment settings train Imagen for 500K steps with a batch size of 512 on 64 TPUv4 chips, using Adam and a cosine learning-rate schedule.
- Training: Imagen models are trained for 500K steps with batch size 512 on 64 TPUv4 chips.These are the reported training-scale settings.
- Optimization: Training uses Adam with a cosine learning-rate schedule and peak learning rate 1e-4.The setup follows the Imagen paper's stated optimizer configuration.
- Preprocessing: Central cropping is removed and images are padded before resizing to 64 × 64 resolution.The stated reason is avoiding misleading crops when captions describe objects outside the crop.
A.2. Text prompts used for evaluation
The evaluation compares two Imagen models on DrawBench counting prompts and a larger prompt set combining number words with CIFAR-10 labels.
- Evaluation setup: Two Imagen models are evaluated: one using official CLIP and one using counting-aware CLIP as the text backbone.The comparison targets the effect of the counting-aware representation.
- Counting prompts: DrawBench contributes 19 counting prompts with object counts ranging from one to five.An example prompt is “Two dogs on the street”.
A.3. Evaluation protocol
The evaluation generates multiple images per counting prompt, then compares generated object counts with requested counts using manual counting and accuracy/MAE measures.
- For each text prompt, 12 images are generated with different random seeds, producing 1296 images overall.
- Generated images are manually evaluated by counting requested object instances and comparing those counts with the prompt.
- The evaluation reports prompt-matching accuracy and mean absolute error between generated and requested object counts.
- Prompts specifying two object counts are successful only when both requested categories have the correct quantities.
A.4. Results
The counting-aware CLIP improves text-conditioned image generation over baseline CLIP, producing requested object counts more reliably and with lower counting error.
- Around 2ˆ better results are achieved by Imagen trained with counting-aware CLIP than by Imagen trained with baseline CLIP.The comparison uses text-conditioned generation evaluation and mean absolute error analyses.
- As requested counts increase, mean absolute errors increase for both counting-aware and baseline models.
- Even when incorrect, the counting-aware model generates object counts closer to the requested number than the baseline.
- Counting-aware CLIP produces specific object counts more reliably than the baseline, which only occasionally generates the correct number.
- Table 6 evaluates prompt matching, while Table 7 reports mean absolute error for generated object counts.
C. Visualization of CountBench benchmark
The CountBench visualization presents diverse, manually verified image-caption examples across counts from two through ten, alongside the prompt-generation and image-generation pipeline.
- CountBench examples vary in image resolution and aspect ratio, while their captions vary in length.
- Examples with larger counts tend to be more grid-like, particularly those labeled nine.
- The prompt-generation pipeline combines each number from two through ten with CIFAR-10 class labels to create textual prompts.
- Imagen generates images from these prompts, which are evaluated using accuracy and mean absolute error.
- Images conditioned on injected numbers are ordered from two-object examples at the top to ten-object examples at the bottom.
- The benchmark visualization includes sampled images labeled with counts from two through ten.