CLIP-Count: Towards Text-Guided Zero-Shot Object Counting

1 INTRODUCTION

• Existing class-agnostic methods require annotated exemplars and can suffer biased counts from high intra-class variation.Reference-less methods avoid exemplar annotation but cannot discriminate the requested object class among multiple classes.Source: §1 · pp. 1–2
• CLIP-Count introduces end-to-end text-guided zero-shot object counting, using natural-language prompts to specify objects without manual exemplar annotations.The approach is presented as a flexible alternative to exemplar-based and reference-less counting.Source: §1 · p. 2
• Text prompts provide more flexible guidance than patch exemplars, spanning general descriptions such as "food" and specific descriptions such as "red apple in a basket".They reduce manual annotation needs during training and testing while allowing different levels of object description.Source: §1 · p. 2
• Natural-language counting remains challenging because prompts can be ambiguous, must localize objects precisely, and require open-vocabulary generalization without large annotated datasets.These constraints make semantic alignment between text and image features non-trivial.Source: §1 · p. 2
• CLIP-Count aligns text with dense visual features through patch-text contrastive learning and propagates semantics across resolutions with hierarchical patch-text interaction.The method learns visual and text prompts while exploiting pretrained CLIP knowledge for density-map generation.Source: §1 · p. 2

2 RELATED WORKS

• Few-shot counting uses exemplars to generalize across object classes, with later methods improving visual backbones, feature extraction, and density prediction.FSC-147 established a dataset for class-agnostic counting, while subsequent work developed stronger architectures and training strategies.Source: §2.1 · p. 2
• Reference-less methods remove exemplar inputs by mining salient objects, but they generally cannot specify which object class should be counted.RCC uses a pretrained ViT to extract salient objects implicitly and directly regress the estimated count.Source: §2.2 · p. 2

2.2 Reference-less and Zero-shot Object Counting

• CLIP-Count combines pretrained CLIP transfer with patch-text alignment, hierarchical multimodal interaction, and CNN decoding for density-map prediction.The overview describes two-resolution multimodal feature maps produced before decoding.Source: Figure 2 · p. 3

2 RELATED WORKS

• CLIP provides an aligned multimodal embedding space that has mainly supported image-level classification, while DenseCLIP extends such representations toward dense prediction.These developments motivate adapting pretrained vision-language features for object counting.Source: §2.3 · p. 3

3.2 Objective

• CLIP-Count estimates a density map for text-specified objects and evaluates on classes disjoint from training classes.The predicted count is obtained by summing the estimated density map.Source: §3.2 · p. 3
• The approach combines visual and text prompt tuning to transfer CLIP’s image-text alignment to dense prediction without relying on exemplar patches.The method distinguishes text-guided zero-shot counting from reference-less counting, which uses only image input.Source: §3.2 · p. 3

3.3 Aligning Text with Dense Visual Features

• The method projects CLIP patch features into the text-embedding dimension and uses patch-text contrastive learning to improve localization.Positive patches are pulled toward the fixed text embedding while negative patches are pushed away.Source: §3.3 · p. 4
• Ground-truth density maps define positive and negative patch sets, enabling contrastive alignment between object-relevant patches and the text prompt.The density map is max-pooled into a patch-level objectness mask before forming the sets.Source: §3.3 · p. 4
• A hierarchical patch-text interaction module propagates text information across image-feature scales to address objects with varying sizes.The module uses cross-attention in a lightweight hierarchical transformer.Source: Figure 4 · p. 4, §3.4 · p. 5

3.4 Adapting CLIP for Density Estimation

• A convolutional decoder progressively upsamples and fuses coarse and fine multimodal feature maps to produce the final density output.The final output uses a 1 × 1 convolution with sigmoid activation.Source: §3.4 · p. 5

4.1 Datasets

• FSC-147 supplies 6,135 images across 147 object classes, with class names, dot supervision, and exemplar boxes, while experiments use only class names as prompts.The exemplar patch annotations are not used.Source: §4.1 · p. 5
• CARPK evaluates cross-dataset transferability using 1,448 bird’s-eye parking-lot images containing 89,777 cars.The dataset is used specifically to test transfer beyond FSC-147.Source: §4.1 · p. 5
• ShanghaiTech contains 1,198 crowd-counting images divided between Parts A and B, whose different collection methods create a cross-part evaluation challenge.Part A has 482 images and Part B has 716 images.Source: §4.1 · p. 5

4.2 Implementation Details

• The architecture uses OpenAI CLIP with a ViT-B/16 backbone, deep visual prompt tuning, two prefix tokens, and a convolutional density decoder.The density decoder uses 3 × 3 convolutions with spatially preserving padding.Source: §4.2 · p. 5
• Training uses 30 contrastive-loss pretraining epochs followed by 200 epochs of mean squared error training on FSC-147.The reported setup uses batch size 32 and AdamW optimization.Source: §4.2 · p. 5

4.3 Evaluation Metrics

• Performance is measured with mean absolute error and root mean squared error, using predicted and ground-truth object counts.Both metrics are computed over the testing images.Source: §4.3 · p. 5

5.1 Quantitative Result

• On FSC-147, CLIP-Count significantly outperforms the state-of-the-art zero-shot counting method while also being compared with few-shot and reference-less methods.The comparison addresses the more challenging requirement of matching text and visual features in zero-shot counting.Source: §5.1 · p. 6, Table 1 · p. 6
• CLIP-Count significantly outperforms the representative reference-less counting method RCC on cross-dataset CARPK evaluation.The model is trained on FSC-147 and evaluated on CARPK without fine-tuning, using the same visual backbone for RCC.Source: §5.1 · p. 6, Table 2 · p. 6
• CLIP-Count outperforms the representative class-specific methods MCNN and CrowdCLIP by 11.2%, 4.0%, 34.3%, and 3.7% under ShanghaiTech cross-dataset evaluation.It also significantly outperforms RCC, demonstrating cross-dataset generalizability without training on ShanghaiTech.Source: §5.1 · p. 6, Table 3 · p. 6

5.2 Qualitative Result

• Qualitative visualizations show high-fidelity density localization and robust counting across object categories, shapes, sizes, and densities.Compared with the prior zero-shot method, CLIP-Count produces more concentrated density patterns and remains competitive with the few-shot LOCA method.Source: Figure 5 · p. 7, §5.2 · p. 8

5.3 Ablation Study

• Ablations show that VPT adds 19.5% MAE and 14.5% RMSE gains with 3% additional parameters, while contrastive loss adds 6.1% and 7.6% improvements.Hierarchical interaction improves handling of object-size variation, and removing contrastive loss can cause visually similar objects to be counted incorrectly.Source: Table 4 · p. 7, §5.3 · p. 8, Figure 6 · p. 8

5.4 Limitations and Future Works

• Text-guidance ambiguity limits CLIP-Count when prompts or queried objects admit multiple interpretations.The paper attributes these cases to coarse class-name annotations in FSC-147 and proposes finer-grained text annotation as future work.Source: §5.4 · p. 8, Figure 7 · p. 8

• CLIP-Count achieves state-of-the-art accuracy and generalizability for zero-shot object counting on FSC-147, CARPK, and ShanghaiTech.Source: §6 · p. 8

• The study ablates the number of resolution scales in the hierarchical patch-text interaction module on FSC-147.Source: A FURTHER ABLATIONS · p. 11, Table 5 · p. 11
• The 1-Scale interaction uses a plain four-layer, four-head ViT to maintain parameter comparability with the 2-Scale interaction.Source: A FURTHER ABLATIONS · p. 11
• The 3-Scale interaction adds the finest 4p × 4p feature-map resolution to the third density-decoder layer.Source: A FURTHER ABLATIONS · p. 11

• Images are resized so the shortest side becomes 224 pixels while preserving aspect ratio.Source: B INFERENCE DETAILS · p. 11
• Inference applies a 224 × 224 sliding window with stride 128 to handle variable image sizes.Source: B INFERENCE DETAILS · p. 11
• Density estimates from overlapping windows are averaged in the overlapped regions.Source: B INFERENCE DETAILS · p. 11

• Qualitative results are reported on FSC-147, CARPK, and ShanghaiTech crowd counting datasets.Source: C ADDITIONAL QUALITATIVE RESULTS · p. 11
• The examples cover objects with different categories, shapes, sizes, and densities.Source: Figure 8 · p. 12
• Figure 8 presents additional qualitative results illustrating CLIP-Count’s generalization across varied object appearances and densities.Source: C ADDITIONAL QUALITATIVE RESULTS · p. 11, Figure 8 · p. 12