Source-linked AI summary
CNN-based Cascaded Multi-task Learning of High-level Prior and Density Estimation for Crowd Counting
Vishwanath A. Sindagi, Vishal M. Patel
TL;DR
Densely crowded scenes make crowd counting difficult because of non-uniform scale variations. The paper introduces an end-to-end cascaded CNN that jointly learns grouped count classification as a high-level prior and density-map estimation, achieving lower count error and better-quality density maps than recent state-of-the-art methods.
Problem
Non-uniform scale and appearance variations make accurate crowd counting and density-map estimation challenging, while existing approaches have limitations in handling these variations and preserving output detail.
Method
An end-to-end cascaded CNN jointly learns ten-group crowd-count classification and density-map estimation, with the prior stage feeding globally relevant features into the density-estimation stage.
Results
The method achieves lower count error and better-quality density maps than recent state-of-the-art methods on challenging public datasets.
Takeaways & Limitations
Jointly learning a coarse count prior and density estimation supports globally discriminative features and full-resolution density-map regression.
Abstract
from arXiv · showhide
Estimating crowd count in densely crowded scenes is an extremely challenging task due to non-uniform scale variations. In this paper, we propose a novel end-to-end cascaded network of CNNs to jointly learn crowd count classification and density map estimation. Classifying crowd count into various groups is tantamount to coarsely estimating the total count in the image thereby incorporating a high-level prior into the density estimation network. This enables the layers in the network to learn globally relevant discriminative features which aid in estimating highly refined density maps with lower count error. The joint training is performed in an end-to-end fashion. Extensive experiments on highly challenging publicly available datasets show that the proposed method achieves lower count error and better quality density maps as compared to the recent state-of-the-art methods.
1. Introduction
The paper addresses joint crowd-count and density-map estimation from single images, where severe perspective distortion creates large scale and appearance variations. It proposes an end-to-end cascaded CNN that learns a high-level count prior jointly with density estimation.
- The paper focuses on jointly estimating crowd count and density maps from a single image.
- Severe perspective distortion causes large variations in crowd-object scale and appearance, making crowd counting difficult.
- The proposed cascade classifies counts into groups to estimate a coarse whole-image count and learn globally relevant discriminative features.
- Shared convolutional layers feed parallel networks for high-level-prior learning and density estimation, whose features are combined to produce high-resolution density maps.
- The method is presented as an end-to-end cascaded CNN for crowd-count classification and density-map estimation.
2. Related work
Earlier crowd-counting methods include detection, regression, density estimation, and CNN-based approaches, but challenges remain in handling scale variation and preserving spatial detail. The paper responds with a cascaded CNN that jointly learns a global prior and density estimation.
- Traditional methods use hand-crafted low-level features mapped to counts or density maps through regression techniques.
- Detection methods are affected by high-density crowds and background clutter, while regression methods map local image-patch features to counts.
- Density-estimation methods preserve spatial information by learning mappings from local patch features to corresponding density maps.
- CNN approaches include layered training, whole-image local and global counting, and architectures designed to address scale variation.
- Existing methods often target selected scales, separately regress count or density, and lose density-map detail through pooling and reduced output resolution.
- The proposed cascaded CNN jointly learns a high-level global prior and density estimation for images with large scale and appearance variations.
3. Proposed method
The proposed method cascades crowd-count classification and density-map estimation in a jointly trained CNN, using the first task as a high-level prior for the second. Shared convolutional features, spatial-pyramid pooling, and fractionally strided convolutions support arbitrary-sized inputs and recover spatial detail.
- Architecture: The network uses two cascaded stages: high-level prior learning followed by density-map estimation, with shared convolutional features.The first stage classifies crowd counts, while the second stage estimates the density map.
- Density estimation stage: The density-estimation stage uses convolutional layers and fractionally strided convolutions to upsample features after pooling-related downsampling.The design addresses detail loss caused by earlier max-pooling layers.
- High-level prior stage: The high-level prior quantizes crowd counts into ten groups to provide coarse global count information despite scale variations.Its classifier uses convolutional and fully connected layers, ending in 10 neurons followed by a sigmoid layer.
- High-level prior stage: Spatial Pyramid Pooling handles arbitrarily sized images before the fully connected layers in the high-level prior stage.SPP aggregates convolutional features into fixed-size outputs, removing the fixed-size constraint imposed by fully connected layers.
- Objective function: The unified objective is L = λLc + Ld, combining high-level-prior classification loss with density-estimation loss.Unlike traditional multi-task learning, the final-stage loss depends on the earlier stage’s output.
- Training and implementation: Training uses augmented arbitrary-sized patches and Gaussian-kernel density maps, with class-weighted classification losses to address class imbalance.Each image produces 300 training patches through random cropping, flipping, and noise addition; density maps sum 2D Gaussian kernels centered on people.
4. Experimental results
Experiments evaluate the proposed method on ShanghaiTech and UCF CC 50 using standard crowd-counting errors and density-map comparisons. The method is reported to achieve lower count error and higher-quality density maps, including the lowest MAE and comparable MSE on UCF CC 50.
- Evaluation setup: Evaluation uses MAE and MSE on the ShanghaiTech and UCF CC 50 datasets.MAE is mean absolute error, while MSE is mean squared error between ground-truth and estimated counts.
- ShanghaiTech dataset: ShanghaiTech experiments compare the proposed method with Zhang et al. and MCNN by Zhang et al..The dataset contains 1,198 annotated images across Parts A and B, with separate training and test sets.
- ShanghaiTech dataset: The proposed method produces lower count error and higher-quality density maps than Zhang et al. and a single-stage CNN on ShanghaiTech.The reported density-map improvement is attributed to fractionally strided convolutional layers.
- Density-map visualization: The reported visualizations present input images, ground-truth density maps, and predicted outputs for ShanghaiTech and UCF CC 50.Figures 3 and 4 organize each example as input, ground truth, and output.
- UCF CC 50 dataset: On UCF CC 50, the proposed network achieves the lowest MAE and a comparable MSE against five recent approaches.Evaluation follows five-fold cross-validation on 50 annotated images with substantial density variation.
5. Conclusions
The paper concludes that a cascaded multi-task CNN jointly learns crowd-count classification and density-map estimation. Its high-level count prior and fractionally strided convolutions support global features and full-resolution density maps, while end-to-end experiments show improvements over recent methods.
- The proposed multi-task cascaded CNN jointly learns crowd-count classification and density-map estimation.
- Count-group classification supplies a high-level prior that helps learn globally relevant discriminative features across large count variations.
- Fractionally strided convolutional layers recover full-resolution density maps after detail loss from earlier max-pooling layers.
- The complete cascade is trained end-to-end and achieves significant improvements on challenging datasets against recent state-of-the-art approaches.