Source-linked AI summary
CLIP-Count: Towards Text-Guided Zero-Shot Object Counting
Ruixiang Jiang, Lingbo Liu, Changwen Chen
TL;DR
Object counting needs guidance that is both class-specific and usable for unseen objects without manual exemplars. CLIP-Count transfers pretrained CLIP to zero-shot counting through dense text-image alignment and hierarchical interaction, achieving state-of-the-art accuracy and generalizability across three datasets, while remaining vulnerable to ambiguous text prompts.
Problem
Existing class-agnostic counting methods require exemplars or cannot specify the target object, while text-guided counting must resolve ambiguous prompts and support open-vocabulary classes.
Method
CLIP-Count aligns text embeddings with dense visual features using patch-text contrastive loss and propagates semantics across resolutions with hierarchical patch-text interaction.
Results
CLIP-Count achieves state-of-the-art accuracy and generalizability on FSC-147, CARPK, and ShanghaiTech crowd counting datasets.
Takeaways & Limitations
Pretrained vision-language models can be adapted end-to-end for text-guided zero-shot object counting and density-map generation.
Takeaways & Limitations
Ambiguous prompts can produce unwanted counts, partly because FSC-147 provides only general class-name annotations.
Abstract
from arXiv · showhide
Recent advances in visual-language models have shown remarkable zero-shot text-image matching ability that is transferable to downstream tasks such as object detection and segmentation. Adapting these models for object counting, however, remains a formidable challenge. In this study, we first investigate transferring vision-language models (VLMs) for class-agnostic object counting. Specifically, we propose CLIP-Count, the first end-to-end pipeline that estimates density maps for open-vocabulary objects with text guidance in a zero-shot manner. To align the text embedding with dense visual features, we introduce a patch-text contrastive loss that guides the model to learn informative patch-level visual representations for dense prediction. Moreover, we design a hierarchical patch-text interaction module to propagate semantic information across different resolution levels of visual features. Benefiting from the full exploitation of the rich image-text alignment knowledge of pretrained VLMs, our method effectively generates high-quality density maps for objects-of-interest. Extensive experiments on FSC-147, CARPK, and ShanghaiTech crowd counting datasets demonstrate state-of-the-art accuracy and generalizability of the proposed method. Code is available: https://github.com/songrise/CLIP-Count.
1 INTRODUCTION
Existing generalized counting methods either require manually annotated exemplars or cannot specify which objects to count. CLIP-Count introduces text-guided zero-shot counting to specify open-vocabulary objects without manual annotations, using CLIP-based dense feature alignment and interaction modules.
- 1 INTRODUCTION: Existing class-agnostic methods require annotated exemplars and can suffer biased counts from high intra-class variation.Reference-less methods avoid exemplar annotation but cannot discriminate the requested object class among multiple classes.
- 1 INTRODUCTION: CLIP-Count introduces end-to-end text-guided zero-shot object counting, using natural-language prompts to specify objects without manual exemplar annotations.The approach is presented as a flexible alternative to exemplar-based and reference-less counting.
- 1 INTRODUCTION: Text prompts provide more flexible guidance than patch exemplars, spanning general descriptions such as "food" and specific descriptions such as "red apple in a basket".They reduce manual annotation needs during training and testing while allowing different levels of object description.
- 1 INTRODUCTION: Natural-language counting remains challenging because prompts can be ambiguous, must localize objects precisely, and require open-vocabulary generalization without large annotated datasets.These constraints make semantic alignment between text and image features non-trivial.
- 1 INTRODUCTION: CLIP-Count aligns text with dense visual features through patch-text contrastive learning and propagates semantics across resolutions with hierarchical patch-text interaction.The method learns visual and text prompts while exploiting pretrained CLIP knowledge for density-map generation.
2 RELATED WORKS
Prior work includes exemplar-guided few-shot counting and reference-less counting, while recent vision-language research motivates transferring image-text representations to dense prediction. CLIP-Count applies this direction to text-guided object counting.
- 2 RELATED WORKS: Few-shot counting uses exemplars to generalize across object classes, with later methods improving visual backbones, feature extraction, and density prediction.FSC-147 established a dataset for class-agnostic counting, while subsequent work developed stronger architectures and training strategies.
- 2 RELATED WORKS: Reference-less methods remove exemplar inputs by mining salient objects, but they generally cannot specify which object class should be counted.RCC uses a pretrained ViT to extract salient objects implicitly and directly regress the estimated count.
- 2.2 Reference-less and Zero-shot Object Counting: CLIP-Count combines pretrained CLIP transfer with patch-text alignment, hierarchical multimodal interaction, and CNN decoding for density-map prediction.The overview describes two-resolution multimodal feature maps produced before decoding.
- 2 RELATED WORKS: CLIP provides an aligned multimodal embedding space that has mainly supported image-level classification, while DenseCLIP extends such representations toward dense prediction.These developments motivate adapting pretrained vision-language features for object counting.
3 METHOD
CLIP-Count adapts CLIP for text-guided zero-shot counting by aligning text with patch-level visual features and decoding hierarchical multimodal representations into density maps.
- 3.2 Objective: CLIP-Count estimates a density map for text-specified objects and evaluates on classes disjoint from training classes.The predicted count is obtained by summing the estimated density map.
- 3.2 Objective: The approach combines visual and text prompt tuning to transfer CLIP’s image-text alignment to dense prediction without relying on exemplar patches.The method distinguishes text-guided zero-shot counting from reference-less counting, which uses only image input.
- 3.3 Aligning Text with Dense Visual Features: The method projects CLIP patch features into the text-embedding dimension and uses patch-text contrastive learning to improve localization.Positive patches are pulled toward the fixed text embedding while negative patches are pushed away.
- 3.3 Aligning Text with Dense Visual Features: Ground-truth density maps define positive and negative patch sets, enabling contrastive alignment between object-relevant patches and the text prompt.The density map is max-pooled into a patch-level objectness mask before forming the sets.
- 3.3 Aligning Text with Dense Visual Features: A hierarchical patch-text interaction module propagates text information across image-feature scales to address objects with varying sizes.The module uses cross-attention in a lightweight hierarchical transformer.
- 3.4 Adapting CLIP for Density Estimation: A convolutional decoder progressively upsamples and fuses coarse and fine multimodal feature maps to produce the final density output.The final output uses a 1 × 1 convolution with sigmoid activation.
4 EXPERIMENTS
Experiments evaluate CLIP-Count across class-agnostic, cross-dataset, and cross-part counting settings using MAE and RMSE, with text prompts and implementation details specified for reproducibility.
- 4.1 Datasets: FSC-147 supplies 6,135 images across 147 object classes, with class names, dot supervision, and exemplar boxes, while experiments use only class names as prompts.The exemplar patch annotations are not used.
- 4.1 Datasets: CARPK evaluates cross-dataset transferability using 1,448 bird’s-eye parking-lot images containing 89,777 cars.The dataset is used specifically to test transfer beyond FSC-147.
- 4.1 Datasets: ShanghaiTech contains 1,198 crowd-counting images divided between Parts A and B, whose different collection methods create a cross-part evaluation challenge.Part A has 482 images and Part B has 716 images.
- 4.2 Implementation Details: The architecture uses OpenAI CLIP with a ViT-B/16 backbone, deep visual prompt tuning, two prefix tokens, and a convolutional density decoder.The density decoder uses 3 × 3 convolutions with spatially preserving padding.
- 4.2 Implementation Details: Training uses 30 contrastive-loss pretraining epochs followed by 200 epochs of mean squared error training on FSC-147.The reported setup uses batch size 32 and AdamW optimization.
- 4.3 Evaluation Metrics: Performance is measured with mean absolute error and root mean squared error, using predicted and ground-truth object counts.Both metrics are computed over the testing images.
5 RESULT AND ANALYSIS
CLIP-Count achieves strong zero-shot and cross-dataset counting performance, while qualitative results show robust localization across varied objects and scenes. Ablations support the contributions of hierarchical interaction, parameter-efficient transfer, and patch-text contrastive alignment, while text ambiguity remains a limitation.
- 5.1 Quantitative Result: On FSC-147, CLIP-Count significantly outperforms the state-of-the-art zero-shot counting method while also being compared with few-shot and reference-less methods.The comparison addresses the more challenging requirement of matching text and visual features in zero-shot counting.
- 5.1 Quantitative Result: CLIP-Count significantly outperforms the representative reference-less counting method RCC on cross-dataset CARPK evaluation.The model is trained on FSC-147 and evaluated on CARPK without fine-tuning, using the same visual backbone for RCC.
- 5.1 Quantitative Result: CLIP-Count outperforms the representative class-specific methods MCNN and CrowdCLIP by 11.2%, 4.0%, 34.3%, and 3.7% under ShanghaiTech cross-dataset evaluation.It also significantly outperforms RCC, demonstrating cross-dataset generalizability without training on ShanghaiTech.
- 5.2 Qualitative Result: Qualitative visualizations show high-fidelity density localization and robust counting across object categories, shapes, sizes, and densities.Compared with the prior zero-shot method, CLIP-Count produces more concentrated density patterns and remains competitive with the few-shot LOCA method.
- 5.3 Ablation Study: Ablations show that VPT adds 19.5% MAE and 14.5% RMSE gains with 3% additional parameters, while contrastive loss adds 6.1% and 7.6% improvements.Hierarchical interaction improves handling of object-size variation, and removing contrastive loss can cause visually similar objects to be counted incorrectly.
- 5.4 Limitations and Future Works: Text-guidance ambiguity limits CLIP-Count when prompts or queried objects admit multiple interpretations.The paper attributes these cases to coarse class-name annotations in FSC-147 and proposes finer-grained text annotation as future work.
6 CONCLUSION
The paper demonstrates that pretrained vision-language models can support end-to-end text-guided zero-shot object counting. CLIP-Count achieves state-of-the-art accuracy and generalizability across three counting datasets.
- CLIP-Count achieves state-of-the-art accuracy and generalizability for zero-shot object counting on FSC-147, CARPK, and ShanghaiTech.
A FURTHER ABLATIONS
The ablation studies vary the number of resolution scales in the hierarchical patch-text interaction module on FSC-147. The results are summarized in Table 5.
- The study ablates the number of resolution scales in the hierarchical patch-text interaction module on FSC-147.
- The 1-Scale interaction uses a plain four-layer, four-head ViT to maintain parameter comparability with the 2-Scale interaction.
- The 3-Scale interaction adds the finest 4p × 4p feature-map resolution to the third density-decoder layer.
B INFERENCE DETAILS
Inference resizes each image so its shortest side is 224 pixels while preserving aspect ratio, then applies overlapping 224 × 224 sliding windows. Predictions in overlaps are averaged.
- Images are resized so the shortest side becomes 224 pixels while preserving aspect ratio.
- Inference applies a 224 × 224 sliding window with stride 128 to handle variable image sizes.
- Density estimates from overlapping windows are averaged in the overlapped regions.
C ADDITIONAL QUALITATIVE RESULTS
Additional qualitative results are shown for FSC-147, CARPK, and ShanghaiTech crowd counting. The examples indicate that CLIP-Count generalizes across object categories, shapes, sizes, and densities.
- Qualitative results are reported on FSC-147, CARPK, and ShanghaiTech crowd counting datasets.
- The examples cover objects with different categories, shapes, sizes, and densities.
- Figure 8 presents additional qualitative results illustrating CLIP-Count’s generalization across varied object appearances and densities.