Source-linked AI summary
Leveraging Unlabeled Data for Crowd Counting by Learning to Rank
Xialei Liu, Joost van de Weijer, Andrew D. Bagdanov
TL;DR
Crowd counting is difficult to train because precise person annotations are laborious and existing labeled datasets are limited. The paper generates rankings from unlabeled crowd imagery and jointly learns ranking with density estimation in a multi-task network. Experiments report significant improvements, state-of-the-art results on ShanghaiTech and UCF CC 50, and strong cross-dataset generalization. The central ranking assumption is that contained crops have no more persons than their enclosing images.
Problem
Crowd counting has limited labeled training data because each person must be meticulously annotated, making training data collection laborious.
Method
The method generates ranked patches from unlabeled crowd images and jointly trains ranking with crowd density estimation in a multi-task network.
Results
The approach obtains state-of-the-art results on ShanghaiTech and UCF CC 50 and generalizes well to other datasets.
Takeaways & Limitations
Unlabeled Internet crowd imagery can be leveraged with annotated data to improve crowd-counting training.
Takeaways & Limitations
The ranking supervision assumes that a crop contained within a larger crop contains the same or fewer persons.
Abstract
from arXiv · showhide
We propose a novel crowd counting approach that leverages abundantly available unlabeled crowd imagery in a learning-to-rank framework. To induce a ranking of cropped images , we use the observation that any sub-image of a crowded scene image is guaranteed to contain the same number or fewer persons than the super-image. This allows us to address the problem of limited size of existing datasets for crowd counting. We collect two crowd scene datasets from Google using keyword searches and query-by-example image retrieval, respectively. We demonstrate how to efficiently learn from these unlabeled datasets by incorporating learning-to-rank in a multi-task network which simultaneously ranks images and estimates crowd density maps. Experiments on two of the most challenging crowd counting datasets show that our approach obtains state-of-the-art results.
1. Introduction
Crowd counting remains difficult and labeled datasets are small because each person requires meticulous annotation. The paper uses ranked sub-images from unlabeled crowd imagery as self-supervision, integrated with supervised density estimation in a multi-task network.
- Crowd counting supports surveillance, safety monitoring, behavior analysis, and other counting applications.
- Perspective distortion, clutter, occlusion, illumination, and scale variation make unconstrained crowd counting extremely difficult.
- Sub-images provide self-supervision because each crop contains the same or fewer persons than its enclosing crowd image.
- Person-counting datasets often contain only a few hundred training images because every individual requires meticulous labeling.
- The method learns ranking and crowd density estimation jointly, rather than relying only on sequential self-supervised pretraining followed by fine-tuning.
- The approach leverages unlabeled crowd imagery and reports state-of-the-art results on two crowd-counting datasets.
2. Related work
Related work spans traditional detectors and CNN-based crowd-counting models, including scale-aware and switching architectures. The paper distinguishes its approach by using automatically ranked patches to exploit unlabeled imagery without side information.
- Traditional approaches: Traditional detection-based approaches fail in extremely dense scenes because of occlusion and low person resolution.
- CNN-based methods: CNN-based methods include basic, multi-column, Hydra-CNN, and switching architectures designed to handle density and scale variation.
- CNN-based methods: Limited labeled data makes it difficult to train deep CNNs without over-fitting, while some methods rely on expensive side information.
- Learning to rank: Prior learning-to-rank studies generally learn ranking functions from ground-truth rankings, unlike this paper’s learning from automatically generated rankings.
- Position relative to prior work: The proposed model uses ranked patches as side information and is described as fast to train, scale-aware, multi-task, and free of side information.
3. Generating ranked image sets for counting
The method converts unlabeled crowd images into ranked patch sets using the containment relation between crops and larger images. It acquires these images through keyword search and query-by-example retrieval, requiring only crowd-scene verification.
- The network compares images and ranks them by person count instead of regressing directly to absolute counts.
- A patch contained within a larger patch must contain the same or fewer persons, creating relative ranks without person-level annotations.
- Only verifying that an image contains a crowd is required before extracting ranked patches according to the proposed procedure.
- Data collection: The keyword dataset contains 1180 high-resolution crowd-scene images collected from Google Images without additional image annotation.
- Data collection: Query-by-example retrieval uses training images as queries and collects the first ten similar images for each query.
4. Learning from ranked image sets
The method combines supervised density-map counting with self-supervised ranking learned from unlabeled crowd-image patches. It explores ranking pretraining, alternating tasks, and simultaneous multi-task training while using VGG-16 features and multi-scale sampling.
- 4.1. Crowd density estimation network: The baseline regresses crowd density maps whose summed values estimate the number of people in each scene.The network is derived from VGG-16 and removes layers that would reduce spatial resolution before adding a convolutional regression layer.
- 4.1. Crowd density estimation network: Pre-trained ImageNet features significantly improve the baseline network’s results.The architecture’s VGG-16 derivation enables reuse of pre-trained features.
- 4.1. Crowd density estimation network: Multi-scale sampling trains on randomly sampled square patches ranging from 56 to 448 pixels instead of whole images.The paper evaluates this sampling strategy as important for good performance.
- 4.2. Crowd ranking network: Ranked patches provide unlabeled training data, replacing the counting loss with average pooling followed by a pairwise ranking loss.Average pooling converts density maps into normalized person-count estimates used for ranking.
- 4.2. Crowd ranking network: The ranking hinge loss penalizes incorrectly ordered estimates, while correctly ordered pairs receive no backpropagated gradient.The margin ε is set to zero, and incorrect pairs update estimates toward the correct ordering.
- 4.3. Combining counting and ranking data: Three combination strategies are evaluated: ranking followed by fine-tuning, alternating-task training, and simultaneous multi-task training.Multi-task training samples ranked and labeled data in each minibatch and weights the counting and ranking losses using λ.
5. Experiments
Experiments evaluate the proposed crowd-counting approach through ablations, data-scaling tests, benchmark comparisons, and transfer learning. The multi-task use of ranked unlabeled imagery improves results across several settings, although performance varies by metric and dataset.
- 5.2. Ablation study: Pre-training reduced UCF CC 50 MAE by around 28%, while adding multi-scale augmentation produced another improvement of around 6%.
- 5.2. Ablation study: Multi-task training reduced UCF CC 50 MAE to 279.6, outperforming ranking plus fine-tuning and alternating-task training.Alternating-task training improved MAE by about 12% over direct fine-tuning, while multi-task training reduced it further.
- 5.2. Ablation study: Using 60% of ShanghaiTech Part A labeled data achieved about the same accuracy as counting-only training with 100% of the labeled data.
- 5.3. Comparison with the state-of-the-art: On UCF CC 50, the Keyword-dataset model reduced MAE from 295.8 to 279.6 versus the state-of-the-art, but had worse MSE and more extreme outliers.The Query-by-example dataset performed slightly worse than the Keyword dataset, possibly because many UCF CC 50 images are low-resolution black-and-white images.
- 5.3. Comparison with the state-of-the-art: On ShanghaiTech, the approach improved performance by about 2% and surpassed the state-of-the-art method on both Parts A and B.The baseline achieved MAE = 77.7 and MSE = 115.9 on Part A, and MAE = 14.7 and MSE = 24.7 on Part B.
- 5.3. Comparison with the state-of-the-art: In transfer learning from ShanghaiTech Part A to UCF CC 50, using only counting information improved MAE by 12% compared with reported results.
6. Conclusions
The method uses the containment relation between nested crops to generate rankings from unlabeled Internet imagery, addressing limited crowd-counting training data. Combined with annotated data in a multi-task network, it achieves strong benchmark and cross-dataset performance.
- A crop contained within a larger crop must contain the same or fewer persons, enabling rankings from unlabeled crowd images.
- The approach exploits abundant Internet training data by automatically generating rankings and combining them with annotated data in a multi-task network.
- The self-supervised task significantly improves results over training only on annotated data.
- The multi-task approach obtains state-of-the-art results on the ShanghaiTech and UCF CC 50 crowd-counting datasets.
- Learned models generalize well to other datasets and significantly outperform the only other reported method on the transfer-learning task.