Source-linked AI summary
Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds
Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Rajpoot, Mubarak Shah
TL;DR
Dense-crowd analysis requires reliable counting, but localization and dataset-quality limitations constrain existing approaches. This paper jointly estimates counts, density maps, and locations through Composition Loss and introduces UCF-QNRF, reporting a unified method and dataset for these tasks. The paper concludes that the tasks are interrelated and that counting benefits from density and localization estimation.
Problem
Dense-crowd counting is important for safety and socio-political assessment, while existing datasets contain low-resolution or erroneous annotations and image-level counts do not reveal error locations.
Method
The method simultaneously estimates counts, density maps, and localization by decomposing their relationships through Composition Loss and sharpening density maps toward localization.
Results
The paper reports that counting benefits from simultaneous density-map estimation and localization, and that UCF-QNRF provides a large high-resolution dataset for all three tasks.
Takeaways & Limitations
The paper supports treating counting, density estimation, and localization as connected tasks and using UCF-QNRF to evaluate them in realistic dense-crowd scenes.
Takeaways & Limitations
A single Gaussian-kernel bandwidth cannot accommodate perspective effects and variable crowd density.
Abstract
from arXiv · showhide
With multiple crowd gatherings of millions of people every year in events ranging from pilgrimages to protests, concerts to marathons, and festivals to funerals; visual crowd analysis is emerging as a new frontier in computer vision. In particular, counting in highly dense crowds is a challenging problem with far-reaching applicability in crowd safety and management, as well as gauging political significance of protests and demonstrations. In this paper, we propose a novel approach that simultaneously solves the problems of counting, density map estimation and localization of people in a given dense crowd image. Our formulation is based on an important observation that the three problems are inherently related to each other making the loss function for optimizing a deep CNN decomposable. Since localization requires high-quality images and annotations, we introduce UCF-QNRF dataset that overcomes the shortcomings of previous datasets, and contains 1.25 million humans manually marked with dot annotations. Finally, we present evaluation measures and comparison with recent deep CNN networks, including those developed specifically for crowd counting. Our approach significantly outperforms state-of-the-art on the new dataset, which is the most challenging dataset with the largest number of crowd annotations in the most diverse set of scenes.
1 Introduction
Dense-crowd counting matters for safety and socio-political assessment, but existing datasets and single-number outputs limit reliable localization and error correction. The paper addresses these gaps with simultaneous counting, density estimation, localization, and the large, diverse UCF-QNRF dataset.
- Motivation: Crowd counting supports safety management and more objective estimates for large gatherings and demonstrations.The paper motivates counting through disaster prevention, political disputes over attendance, and the limits of human numerical cognition.
- Approach: The proposed method jointly estimates counts, density maps, and person locations by exploiting their decomposition relationships.Density maps can be sharpened toward localization maps, whose integral should equal the true count.
- Motivation: Localization provides dot locations that let analysts correct false positives and false negatives, enabling precise counts when reliable image-level counts are needed.This is especially useful when only a few image samples are available.
- Dataset: UCF-QNRF contains high-resolution, realistic scenes with diverse viewpoints, densities, lighting, and count ranges for dense-crowd analysis.The dataset includes background regions and uncropped environments, making it more realistic and difficult than datasets focused only on cropped crowds.
2 Related Work
Prior crowd-counting research spans regression, density-map estimation, and localization, but methods often address only one output or use separate stages. CNN-based methods learn image features for density prediction, while localization approaches use density maps to guide detection or optimization.
- Task landscape: The related work reflects three connected task formulations: counting-by-regression, density-map estimation, and localization.The paper’s approach builds on this task landscape by treating the outputs as interrelated rather than isolated.
- Counting: Regression-based methods produce image-level counts but cannot directly provide density maps or person localization.Earlier approaches also relied on hand-crafted features for low resolution, perspective distortion, and occlusion.
- Density estimation: CNN approaches predict density maps using patch aggregation, normalized losses, multiscale filters, or fused global and local context.These methods differ in how they handle overlapping patches, density consistency, scale variation, and contextual information.
- Localization: Localization methods use density maps as regularizers or inputs to optimization for selecting object locations in crowded scenes.Reported strategies include detection regularization and integer programming over sliding-window density maps.
3 Deep CNN with Composition Loss
The method jointly estimates count, density, and localization maps by decomposing their interrelated objectives into a composition loss. Adaptive Gaussian maps become progressively sharper, while integrated map outputs provide additional count constraints and intermediate supervision.
- Composition Loss: The network simultaneously estimates count, density, and localization maps by decomposing the loss of these interrelated tasks.The architecture branches from a base DenseNet into multiple dense blocks for the map predictions.
- Adaptive Gaussian Kernel: Adaptive bandwidth σ_i uses each person’s nearest-neighbor distance, capped by a maximum threshold τ, to preserve location information across perspective and density changes.A single bandwidth can separate nearby people poorly or blur other regions excessively.
- Density and Localization Maps: The function f_k(σ) = σ^1/k generates progressively sharper density maps, with k → ∞ approaching a binary localization map.The experiments use three density levels, with the final level serving as the localization map.
- Composition Loss: Each estimated density or localization map is integrated to produce a count, and the final count can average these map counts with the base-network regression.Training uses mean-square error for the maps and Euclidean loss for regression, combined as a weighted mean of four losses.
- DenseNet with Composition Loss: DenseNet-201 supplies the base network, comprising four dense blocks with {6, 12, 48, 32} convolutional-layer sets.The implementation uses Adam with an initial learning rate of 0.001 for 70 epochs and batch size 16.
4 The UCF-QNRF Dataset
UCF-QNRF is constructed from Flickr, web search, and Hajj footage to provide diverse, high-density crowd imagery. Manual pruning, annotation, and verification produced 1,535 images with over 1.25 million annotations.
- Dataset Collection: Images came from Flickr, Google Image Search, and Hajj footage selected to vary locations, viewpoints, perspective effects, and time of day.Search queries covered crowds, Hajj, spectators, pilgrimages, protests, and concerts.
- Initial Pruning: Images were pruned for low density, non-human objects, blur, low resolution, excessive perspective effects, watermarks, or text occupying over 10% of the image.High-density images were retained when heads remained visually separable for annotation.
- Annotation: Annotation used two stages: 14 annotators marked images, then 4 verifiers corrected annotation errors.The process required 2,000 human-hours, including 1,300 annotator hours and 200 verifier hours.
- Statistics: 1,535 images contain 1,251,642 annotations, split into 1,201 training and 334 test images.Images were sorted by absolute count and every fifth image was assigned to the test set; counts range from 49 to 12,865.
5 Definition and Quantification of Tasks
The paper evaluates crowd analysis through counting and localization measures. Counting estimates the number of people in an image, while localization uses matched predicted and ground-truth points to compute precision-recall performance.
- Counting: Counting estimates the number of people in a crowd image, but does not provide their locations or spatial distribution.The paper notes that automated counts and densities across multiple images can support integration over a larger crowd-covered area.
- Localization: Localization error uses greedy one-to-one matching between estimated and ground-truth locations, followed by precision and recall at distance thresholds from 1 to 100 pixels.Overall localization performance is summarized by the area under the precision-recall curve, L-AUC.
6 Experiments
Experiments evaluate the proposed approach on counting, density map estimation, localization, and ablations of its network and Composition Loss design. Results show strong performance across tasks, with density estimation and localization outperforming compared methods and multiple density levels improving ablation results.
- 6 Experiments: The experiments evaluate counting, density map estimation, localization, and an ablation study of the proposed architecture and Composition Loss.Counting uses UCF-QNRF and compares the approach with general deep networks and crowd-counting methods.
- 6.2 Density Map Estimation: The proposed approach outperforms existing density map estimation methods by an order of magnitude on Histogram intersection results.The comparison includes MCNN, SwitchCNN, and CMTL, which directly regress crowd density during training.
- 6.3 Localization: The proposed approach is superior to all compared methods for localization, evaluated using precision, recall, and AUC averaged over four distance thresholds.Predicted locations are obtained by thresholding local peaks and matched one-to-one with ground-truth locations.
- 6.4 Ablation Study: Multiple density levels outperform using Density Level 1 alone, while adding a second density level improves results slightly over a single level.The ablation compares D1 only with D1 and D2 only and excludes D∞ in the latter configuration.
- 6.4 Ablation Study: The Density Network contributes significantly across the three tasks, and branching from middle base-network layers performs better than the evaluated alternatives.Equalizing counts from all sources slightly worsens counting but improves density and localization maps.
7 Conclusion
The paper presents Composition Loss for jointly estimating counts, density maps, and localizations in dense crowds, alongside the UCF-QNRF dataset. Experiments and ablations demonstrate the approach’s performance, while the dataset supports future crowd-analysis research.
- Composition Loss decomposes the interrelated counting, density-map estimation, and localization tasks so they can train a neural network simultaneously.
- Counting performance benefits from jointly estimating density maps and localizations.
- UCF-QNRF is a large-scale, high-resolution dataset designed for training and evaluating all three dense-crowd analysis tasks.
- Experiments with recent deep architectures and a detailed ablation study show that the proposed approach achieves good performance.
- The dataset is intended to support applications in safety and surveillance, public-infrastructure planning, and assessing the political significance of crowd events.