Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds

1 Introduction

• Computer-vision crowd counts offer fast, objective estimates for safety and political contexts and can extend to cells, bacteria, wildlife, and vehicles.Crowd estimates matter for both passive gatherings, where disasters can occur, and active protests where opposing sides may claim divergent numbers.Source: §1 · pp. 1–2
• The approach jointly estimates counts, density maps, and locations, treating each task as a special case of another and sharpening density maps toward localization whose integral equals the true count.The authors note that localization is difficult to estimate directly because it is sparse, while density maps provide weaker location information and counts provide none.Source: §1 · p. 2
• Density maps and localization support tracking initialization and let analysts correct false positives and negatives to obtain precise counts when reliable estimates are needed from few images.A single image-level count does not identify the source of error; localization provides candidate dot locations for manual review.Source: §1 · pp. 2–3
• Earlier datasets had limitations for dense-crowd analysis: low-density or single-scene data, and low-resolution UCF annotations that could omit people or misplace counts.UCF-QNRF is introduced to address the annotation problems illustrated for existing datasets.Source: Figure 1 · p. 2, §1 · p. 3
• UCF-QNRF spans diverse viewpoints, crowd densities, and lighting, with realistic backgrounds that make it more realistic and difficult than crowd-only crops.Its average people-per-pixel density is lowest, partly because images include background regions alongside high- and zero-density areas.Source: §1 · p. 3
• Compared with UCF CC 50, UCF-QNRF is 20 times larger in number of images and annotations, respectively, and has the most high-count crowd images and annotations among the compared datasets.Source: §1 · pp. 3–4

• Earlier regression approaches mapped global or local-patch features to counts,,,, but could not estimate density maps or localize people.Hand-crafted features, sometimes combined,, addressed low resolution, perspective distortion, and severe occlusion; CNNs learn multiple feature maps automatically.Source: §2 · p. 4
• Zhang et al. alternated between predicting density maps and patch counts, while Lebanoff and Idrees used normalized Euclidean loss for consistent counting across densities.Source: §2 · pp. 4–5
• Other CNN methods used multi-scale filters, global and local context, or a learned switching mechanism to route patches to CNN regressors.Source: §2 · p. 5

3.1 Composition Loss

• Three density levels progressively sharpen an adaptive Gaussian density map into a localization map, preserving head locations despite perspective and crowd-density variation.Each person’s kernel bandwidth is the minimum of the distance to the nearest neighbor or a maximum threshold, τ; the experiments use three density levels, with the last being localization.Source: §3.1 · pp. 5–6
• The weighted-mean loss combines map MSE losses with a count regression loss, while the mean of four predicted counts is trained to equal the ground-truth count.Counts come from the base CNN output and the integrals of the three density/localization maps.Source: §3.1 · p. 7

3.2 DenseNet with Composition Loss

• The base network is DenseNet-201, with four dense blocks containing 6, 12, 48, and 32 sets of 1 × 1 and 3 × 3 convolutional layers.Training uses Adam for 70 epochs with batch size 16 and an initial learning rate of 0.001, halved every 20 epochs.Source: §3.2 · p. 7
• The density network branches from DenseBlock2, concatenating prior features and predicted density levels to produce density and localization maps.Density Level 1 is computed directly from DenseBlock2 features; the proposed architecture and additional count-consistency constraints are illustrated in Fig. 3.Source: Figure 3 · p. 6, §3.2 · p. 7

Dataset Collection

• Images came from Flickr, Web Search, and Hajj footage, with varied Hajj locations, viewpoints, perspective effects, and times of day.Flickr and Google Image Search queries targeted crowd-related scenes; duplicate images were checked using image similarity and manual verification.Source: §4 · pp. 7–8

Initial Pruning

• Images were pruned for unsuitable content or quality, and retained dense-crowd images were cropped to remove regions with unrecognizable or missing head annotations.Pruning excluded non-crowds or low-density scenes, non-human objects, blur or low resolution, strong high-angle perspective, watermarks, and images with text occupying more than 10%.Source: §4 · p. 8

Annotation

• Fourteen annotators and four verifiers completed a two-stage annotation process that used 2, 000 human-hours.Annotators marked images first, then verifiers corrected annotation errors.Source: §4 · p. 8

Statistics

• The dataset contains 1, 535 images and 1, 251, 642 annotations, split into 1201 training and 334 test images.Counts range from 49 to 12, 865 people per image, with a median of 425 and mean of 815.4; every fifth image after sorting by count was assigned to test.Source: §4 · p. 8

Counting

• Counting estimates the number of people in an image and is evaluated with C-MAE, C-MSE, and C-NAE.For large-crowd estimates, Jacob’s Method divides an area into sections, estimates their densities, and extrapolates the mean density across the full area; image density is people per ground area.Source: §5 · p. 9

Density Map Estimation

• Density-map estimation predicts per-pixel density to preserve spatial distribution, revealing dangerous local concentrations that total counts can conceal.It uses per-pixel DM-MAE and DM-MSE, plus normalized-map DM-HI, which emphasizes distribution error while discarding absolute-count differences.Source: §5 · p. 9

Localization

• Directly detecting people is infeasible in dense crowds because of severe occlusion and low pixel counts for people farther from the camera.Although most counting methods therefore use direct regression, precise locations can be needed to initialize tracking in very high-density videos.Source: §5 · p. 9
• Localization error is measured by greedily matching estimated and ground-truth positions one-to-one, then computing L-AUC from precision and recall across 1–100-pixel thresholds.The thresholds are evaluated at each pixel distance from 1 through 100 pixels.Source: §5 · p. 9

6.1 Counting

• The proposed method significantly outperforms competing deep CNNs and crowd-counting approaches on counting.Networks were trained on randomly sampled 448-, 224-, and 112-pixel patches, with higher sampling probability for higher-count regions; test images were evaluated in zero-padded 224 × 224 cells whose counts were aggregated.Source: Table 3 · p. 10, §6.1 · p. 10

6.2 Density Map Estimation

• The proposed approach outperforms existing density-estimation methods by an order of magnitude on Histogram intersection.Comparisons include MCNN, SwitchCNN, and CMTL, which are designed for crowd-density estimation.Source: Table 4 · p. 11, §6.2 · p. 11

6.3 Localization

• The proposed approach is superior to the compared methods for localization; DenseNet and Encoder-Decoder outperform ResNet and MCNN.The method extracts local peaks using threshold-based non-maximal suppression, matches predictions one-to-one with ground truth at distance thresholds, and computes precision and recall. Localization performance depends on post-processing, which the authors identify as an important direction for further research.Source: §6.3 · p. 11, Table 5 · p. 12

6.4 Ablation Study

• Using D1 alone performs worse than the proposed multi-level Composition Loss, while adding D2 to D1 improves results slightly.The ablation compares density levels and includes D∞, the localization-map estimate, alongside D2 and D1.Source: §6.4 · pp. 12–13, Table 6 · p. 13
• The Density Network contributes significantly across the three tasks, and branching it from middle base-network layers performs better than branching from later layers.Enforcing equal counts from all sources slightly worsens counting but helps estimate density and localization maps; Table 6's decreasing error rates from right to left indicate a positive influence from Composition Loss.Source: §6.4 · p. 13

7 Conclusion

• Composition Loss decomposes the interrelated tasks of counting, density-map estimation, and localization to train a neural network that solves them simultaneously.The authors report that counting performance benefits from density-map estimation and localization.Source: §7 · p. 14
• UCF-QNRF is a large-scale dense-crowd dataset curated with high-resolution images for counting, density-map estimation, and localization.The paper describes its collection and annotation process.Source: §7 · p. 14
• Experiments using several recent deep architectures and a detailed ablation study show that the proposed approach achieves good performance.Source: §7 · p. 14
• The authors identify potential dataset applications in safety and surveillance, public-infrastructure design and expansion, and gauging the political significance of crowd events.Source: §7 · p. 14