Source-linked AI summary
CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model
Dingkang Liang, Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, Xiang Bai
TL;DR
Crowd counting relies on costly point-level labels, while prior pure-unsupervised methods leave a performance gap. CrowdCLIP transfers CLIP to counting with ranking-based image-text fine-tuning and progressive filtering. It outperforms the prior unsupervised state of the art and surpasses some fully supervised methods under cross-dataset evaluation.
Problem
Point-level crowd-counting labels are costly, and the applicability of vision-language models to counting had not been explored.
Source: §1 · pp. 1–2Method
CrowdCLIP fine-tunes CLIP with ranking text prompts matched to size-sorted patches, then progressively filters patches and maps them to count intervals.
Source: §1 · p. 2, §4.1 · p. 3Results
CrowdCLIP outperforms CSS-CCNN by 35.2% on UCF-QNRF in MAE and surpasses some fully supervised works under cross-dataset validation.
Source: §1 · p. 2Takeaways & Limitations
CrowdCLIP demonstrates that vision-language knowledge can be transferred to unsupervised crowd counting across five challenging datasets.
Source: §1 · p. 2, §7 · p. 8CrowdCLIP provides count-level information for an image, not point-level localization.
Source: §6.4 · p. 8
Abstract
from arXiv · showhide
Supervised crowd counting relies heavily on costly manual labeling, which is difficult and expensive, especially in dense scenes. To alleviate the problem, we propose a novel unsupervised framework for crowd counting, named CrowdCLIP. The core idea is built on two observations: 1) the recent contrastive pre-trained vision-language model (CLIP) has presented impressive performance on various downstream tasks; 2) there is a natural mapping between crowd patches and count text. To the best of our knowledge, CrowdCLIP is the first to investigate the vision language knowledge to solve the counting problem. Specifically, in the training stage, we exploit the multi-modal ranking loss by constructing ranking text prompts to match the size-sorted crowd patches to guide the image encoder learning. In the testing stage, to deal with the diversity of image patches, we propose a simple yet effective progressive filtering strategy to first select the highly potential crowd patches and then map them into the language space with various counting intervals. Extensive experiments on five challenging datasets demonstrate that the proposed CrowdCLIP achieves superior performance compared to previous unsupervised state-of-the-art counting methods. Notably, CrowdCLIP even surpasses some popular fully-supervised methods under the cross-dataset setting. The source code will be available at https://github.com/dk-liang/CrowdCLIP.
1. Introduction
CrowdCLIP frames unsupervised crowd counting as image-text matching to avoid annotation costs, using ranking-based fine-tuning and progressive filtering. It outperforms prior unsupervised methods and surpasses some fully supervised methods in cross-dataset evaluation.
1. Introduction
- Crowd-counting density models require point-level annotations assigning a point to each human head, an expensive, laborious process; weakly and semi-supervised alternatives still incur considerable labeling costs.NWPU-Crowd [49] contains 5,109 images and required 30 annotators and 3,000 human hours for annotation.Source: §1 · pp. 1–2
- Zero-shot CLIP performs unsatisfactorily because it is not tuned for crowd semantics and ambiguous patches may contain no human heads.Original CLIP is mainly trained to recognize single-object images.Source: §1 · p. 2
- CrowdCLIP treats counting as image-text matching and fine-tunes the image encoder with ranking text prompts and a multimodal ranking loss on size-sorted patches.The prompts guide the encoder to capture crowd semantics.Source: §1 · p. 2
- At test time, progressive filtering selects highly related crowd patches and maps them to appropriate count intervals, reducing the impact of ambiguous patches.The first two stages use coarse-to-fine classification; the final stage maps selected patches to a count.Source: §1 · p. 2
- CrowdCLIP outperforms CSS-CCNN by 35.2% on UCF-QNRF in MAE and surpasses some fully supervised works under cross-dataset validation.Experiments span five challenging datasets.Source: §1 · p. 2
2. Related Works
Prior crowd-counting work includes density-map and localization approaches, methods that reduce but do not eliminate annotation needs, and vision-language transfer to other tasks. Pure unsupervised counting had limited prior work, motivating CrowdCLIP's image-text matching approach.
2.1. Fully-Supervised Crowd Counting
- Fully supervised counting commonly regresses density maps generated from labeled point maps, while other approaches use multi-scale features, scaling, attention, interval classification, or localization.The related work describes localization methods that predict boxes or maps, or directly regress point coordinates.Source: §2.1 · p. 2
2.2. Weakly-/Semi-/Unsupervised Crowd Counting
- Weakly supervised methods use count-level labels and semi-supervised methods combine a small labeled set with unlabeled data, but both still require annotations.CSS-CCNN is described as the only prior pure-unsupervised method; it uses a power-law crowd prior, while CrowdCLIP casts counting as image-text matching.Source: §2.2 · p. 3
2.3. Vision-Language Contrastive Learning
- Vision-language pretraining has been transferred to tasks including object detection, semantic segmentation, and generation, and this paper studies its transfer to unsupervised crowd counting.Source: §2.3 · p. 3
3. Preliminary
The paper sets unsupervised crowd counting as estimating pedestrian numbers without annotated training images, while manually labeled validation or test data may be used for evaluation. It builds on CLIP's image-text semantic similarity and studies extending that correspondence to counting.
3.1. A Revisit of CLIP
- CLIP encodes images and text separately, computes their semantic similarity, and supports zero-shot or open-vocabulary classification using text prompts and class names.Source: §3.1 · p. 3
- The paper studies extending CLIP's image-text correspondence to crowd counting, which it identifies as the first such investigation.Source: §3.1 · p. 3
3.2. Unsupervised Crowd Counting
- The paper defines unsupervised counting as training without any annotated images while allowing supervision from the original data; manually annotated validation or test sets are used only for evaluation.Source: §3.2 · p. 3
4. Our Method
CrowdCLIP adapts CLIP to unsupervised crowd counting through ranking-based image-encoder fine-tuning and progressive filtering at inference. Training aligns size-ordered patches with count prompts, while testing filters likely crowd patches before mapping them to count intervals.
4.1. Ranking-based Contrastive Fine-tuning
- Count-interval prompts describe size-ordered patches, guiding image-encoder fine-tuning with a multimodal ranking loss while the text encoder remains frozen.The approach does not require labeled crowd images, unlike the cited prior ranking methods [26] [27].Source: §4.1 · pp. 3–5
- Training crops patches in size order, gives them a shared center, and resizes them to a common input size before image encoding.The ordering premise is that later patches contain at least as many heads as earlier ones.Source: §4.1 · p. 3
- Ranking prompts use “There are [class] persons in the crowd,” with class counts spaced by a reference count and interval to preserve ordinal relationships in language space.The method sets the number of patches equal to the number of text ranks to make the similarity matrix square.Source: §4.1 · p. 4
4.2. Progressive Filtering Strategy
- Three-stage inference uses the original encoder to find crowd patches, filters for human heads, then uses the fine-tuned encoder to select a count interval.The text prompts can be encoded in advance and reused during inference.Source: §4.2 · p. 5
5. Experiments
Experiments cover five crowd-counting datasets and evaluate performance with MAE and MSE. The implementation uses CLIP with a ViT-B/16 backbone and specifies training and inference settings.
5.1. Dataset and Evaluation Metric
- The experiments cover UCF-QNRF, JHU-Crowd++, ShanghaiTech Parts A and B, and UCF-CC50, using MAE and MSE for evaluation.The metric definitions use estimated and ground-truth counts over the test images.Source: §5.1 · p. 5
5.2. Implement Details
- The implementation fine-tunes CLIP’s ViT-B/16 image encoder for 100 epochs with frozen text-encoder weights, RAdam at 1e-4, and six image patches and text ranks.Ranking prompts use reference count 20 and interval 35; the inference grid size varies by dataset.Source: §5.2 · p. 5
- Figure 3’s first row comes from ShanghaiTech Part A, and its second row comes from UCF-QNRF.Source: Figure 3 · p. 6
6. Results and Analysis
CrowdCLIP outperforms the prior unsupervised state of the art across evaluated datasets and shows competitive or better results than some fully supervised methods in cross-dataset evaluation. Ablations examine progressive filtering, encoder fine-tuning, prompt design, patch count, and training-data size; the method’s stated limitation is that it provides counts but not localization.
6.1. Comparison with the State-of-the-Arts
- On UCF-QNRF, CrowdCLIP improves over CSS-CCNN by 35.2% in MAE and 32.3% in MSE, and reports 213.7 MAE and 576.1 MSE on JHU-Crowd++.The authors also report that CrowdCLIP surpasses CSS-CCNN across all evaluated datasets and performs well in scenes with different scales.Source: §6.1 · p. 6
- On UCF-QNRF, CrowdCLIP’s MAE is 283.3 versus 277.0 for MCNN, and it outperforms method on ShanghaiTech Part A and UCF CC 50.These comparisons are with popular fully supervised methods.Source: §6.1 · p. 6
6.2. Cross-Dataset Validation
- For ShanghaiTech Part A→Part B transfer, CrowdCLIP beats MCNN by 18.3% in MAE and D-ConvNet by 18.6% in MSE; it also improves over CSS-CCNN by 37.6% and 37.1% in MAE on the two stated transfers.The latter improvements are for Part A→UCF-QNRF and UCF-QNRF→Part A, respectively; CrowdCLIP achieves the best results in the other transfer settings as well.Source: Table 2 · p. 7, Table 3 · p. 7, §6.2 · p. 7
6.3. Ablation Study
- On UCF-QNRF, progressive inference improves original CLIP from 528.7 to 414.7 MAE, and using the fine-tuned image encoder for the final stage brings a further significant gain.The original CLIP’s 528.7 MAE is worse than CSS-CCNN’s 437.0; the authors attribute the further gain to ranking-based contrastive fine-tuning.Source: §6.3 · p. 7, Table 4 · p. 8
- Prompts with R0=20 and K=35 perform best, while abstract prompts also perform well; fine-tuning the image encoder improves MAE from 414.7 to 283.3, whereas tuning the text encoder degrades performance.The authors suggest that text-encoder fine-tuning breaks language priors, while a fixed text encoder provides a reference for mapping image features into ranking language space.Source: §6.3 · p. 7, Table 5 · p. 8, Table 6 · p. 8
- MAE is best with 4 patches and MSE with 5, while performance improves steadily with more training data and can benefit from extra data.The data-size comparison uses ShanghaiTech Part A as extra data; the supplied figure caption identifies UCF-QNRF as the evaluation dataset.Source: §6.3 · pp. 7–8, Table 7 · p. 8, Figure 4 · p. 8
6.4. Limitation
- CrowdCLIP provides count-level information only; the authors identify unsupervised localization as future work.Point-level information is described as useful for crowd analysis.Source: §6.4 · p. 8
7. Conclusion
CrowdCLIP transfers CLIP to unsupervised crowd counting through image-encoder fine-tuning and progressive filtering. Experiments on challenging datasets demonstrate state-of-the-art performance.
7. Conclusion
- CrowdCLIP transfers knowledge from the vision-language pre-trained model CLIP to unsupervised crowd counting.The authors present it as a new framework for the task.Source: §7 · p. 8
- The framework fine-tunes the image encoder through multi-modal ranking loss and uses progressive filtering.Source: §7 · p. 8
- Experiments on challenging datasets demonstrate state-of-the-art performance for CrowdCLIP.The authors hope the method provides a new perspective for crowd counting.Source: §7 · p. 8
A. Inference speed
CrowdCLIP has dynamic inference speed because progressive filtering selects different numbers of crowd patches across images. Its reported speed is competitive with CSS-CCNN, while high-resolution feature requirements slow fully supervised methods.
A. Inference speed
- CrowdCLIP's inference speed ranges from 24.0 to 50.8 FPS, depending on the number of selected crowd patches.The lower endpoint applies when all image patches contain human heads; the upper endpoint applies when no crowd patches are present.Source: A. Inference speed · p. 12, Table 1 · p. 12
- CrowdCLIP is highly competitive in inference speed with the unsupervised state-of-the-art method CSS-CCNN.Source: A. Inference speed · p. 12
- Fully supervised methods [18] [31] have low inference speed because they require high-resolution features to generate density maps.Source: A. Inference speed · p. 12
B. More visualizations
The visual comparisons contrast zero-shot CLIP with CrowdCLIP, including a Seoul Halloween crowd-crush evaluation without ground truth. CrowdCLIP's predictions are close to CSRNet's in most of those scenes.
B. More visualizations
- In the qualitative comparison, zero-shot CLIP does not understand crowd semantics well, while CrowdCLIP generates more reasonable attention and performs better.Figure 1 presents ground truth, zero-shot CLIP results, and CrowdCLIP results from left to right; zero-shot CLIP uses the original non-fine-tuned CLIP image encoder.Source: B. More visualizations · p. 12, Figure 1 · p. 13
- On the Seoul Halloween crowd-crush scenes, CrowdCLIP's predictions are close to CSRNet's in most cases, but no ground truth is available.The compared models were trained on UCF-QNRF; CSRNet is a representative fully supervised method.Source: B. More visualizations · p. 12, Figure 2 · p. 13