Source-linked AI summary
CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes
Yuhong Li, Xiaofan Zhang, Deming Chen
TL;DR
Highly congested scenes require both accurate counting and spatially informative density maps, while existing multi-column approaches introduce training and redundancy concerns. CSRNet uses a pure-convolutional network with a CNN front-end and dilated-convolution back-end to generate density maps. It achieves state-of-the-art results across crowd datasets and extends effectively to vehicle counting.
Problem
Highly congested-scene analysis needs accurate counts and spatially coherent density maps because identical crowd counts can represent different distributions.
Method
CSRNet combines the first 10 VGG-16 convolutional layers as a front-end with dilated convolutional layers that enlarge receptive fields without losing resolution.
Results
CSRNet delivers state-of-the-art performance across four crowd-counting datasets and achieves 47.3% lower MAE than CP-CNN on ShanghaiTech Part B.
Takeaways & Limitations
The pure-convolutional architecture supports end-to-end training, high-quality density-map generation, and extension from crowd counting to vehicle counting.
Takeaways & Limitations
Prior multi-column approaches require more training time, contain redundant branches, and depend on difficult-to-define density-level classification.
Abstract
from arXiv · showhide
We propose a network for Congested Scene Recognition called CSRNet to provide a data-driven and deep learning method that can understand highly congested scenes and perform accurate count estimation as well as present high-quality density maps. The proposed CSRNet is composed of two major components: a convolutional neural network (CNN) as the front-end for 2D feature extraction and a dilated CNN for the back-end, which uses dilated kernels to deliver larger reception fields and to replace pooling operations. CSRNet is an easy-trained model because of its pure convolutional structure. We demonstrate CSRNet on four datasets (ShanghaiTech dataset, the UCF_CC_50 dataset, the WorldEXPO'10 dataset, and the UCSD dataset) and we deliver the state-of-the-art performance. In the ShanghaiTech Part_B dataset, CSRNet achieves 47.3% lower Mean Absolute Error (MAE) than the previous state-of-the-art method. We extend the targeted applications for counting other objects, such as the vehicle in TRANCOS dataset. Results show that CSRNet significantly improves the output quality with 15.4% lower MAE than the previous state-of-the-art approach.
1. Introduction
Congested-scene analysis must represent spatial crowd distributions, not only total counts, because identical counts can correspond to different scenes. CSRNet addresses this need with a deeper convolutional density-map generator and reports lower MAE than prior methods across several datasets.
- Motivation: Identical crowd counts can correspond to different spatial distributions, making density maps more informative than counts alone for high-risk decisions.Figure 1 illustrates three ShanghaiTech Part B images containing 95 people but having different spatial arrangements.
- Motivation: Pixel-by-pixel density prediction requires spatially coherent maps with smooth transitions between neighboring pixels.Irregular crowd clusters and varying camera perspectives make this prediction difficult.
- Limitations of prior approaches: Multi-scale architectures can require substantial training time and introduce ineffective branching as networks become deeper.The introduction identifies MCNN as an example of this branch-structure problem.
- Limitations of prior approaches: MCNN columns with small, medium, and large kernels show similar estimated-error patterns across 50 ShanghaiTech Part A test cases, suggesting redundant learned features.This behavior conflicts with the intended role of separate columns learning different features for different congestion levels.
- Contribution: CSRNet uses a deeper CNN-based density-map generator and reports lower MAE than the previous state-of-the-art CP-CNN across ShanghaiTech, UCF CC 50, and WorldExpo’10.The reported reductions are 7%, 47.3%, 10.0%, and 2.9% for ShanghaiTech Part A, Part B, UCF CC 50, and WorldExpo’10, respectively; UCSD performance is 1.16 MAE.
2. Related work
Related work spans detection-, regression-, density-estimation-, and CNN-based approaches, with CNN methods improving congested-scene analysis. The paper highlights redundancy, training cost, and classification overhead in multi-column CNN designs, motivating a simpler alternative.
- Approach categories: Crowd-scene methods are commonly classified as detection-based, regression-based, or density-estimation-based approaches.CNN-based solutions are described as outperforming traditional methods in this task.
- Detection-based methods: Detection-based methods perform poorly in highly congested scenes because targeted people are often obscured.Some methods instead detect body parts rather than complete bodies.
- Regression-based methods: Regression-based methods learn feature-to-count relations from cropped patches, but local saliency can be overlooked and produce inaccurate regional results.Later work learns mappings to object density maps to incorporate local saliency.
- CNN-based methods: CNN-based research includes patch-based, end-to-end, selective-sampling, boosting, and density-map prediction methods.These approaches build on CNN success in classification and recognition.
- Limitations of multi-column CNNs: Multi-column methods such as Switch-CNN and Contextual Pyramid CNN achieve state-of-the-art results but are difficult to train and can contain redundant branches.Their designs also use density-level classifiers to route input patches among regressors or columns.
- Limitations of multi-column CNNs: Multi-column designs allocate parameters to density-level classification and may leave fewer resources for final density-map generation.Fine-grained density classification also increases the number of columns and design complexity.
3. Proposed Solution
CSRNet combines a VGG-16 convolutional front-end with a dilated-convolution back-end to generate density maps while preserving resolution and enlarging receptive fields. Its configuration uses small convolutional kernels, selected pooling, geometry-adaptive ground-truth maps, patch augmentation, and end-to-end training.
- Network architecture: CSRNet uses the first 10 VGG-16 convolutional layers as a front-end and dilated convolutional layers as a back-end for density-map generation.The model uses pure convolutional layers and supports input images with flexible resolutions.
- Dilated convolution: Dilated convolution enlarges receptive fields without increasing parameters or computation while preserving feature-map resolution.A k × k kernel becomes k + (k −1)(r −1) with dilation stride r, enabling multi-scale contextual aggregation.
- Dilated convolution: Compared with pooling, convolution, and upsampling, dilation with factor = 2 produces an output matching the input dimension and retains more detailed information.The alternative scheme requires downsampling and subsequent upsampling, whereas dilated convolution does not require pooling or deconvolutional layers.
- Network configuration: The selected architecture keeps three pooling layers and the first ten VGG-16 layers to balance output accuracy with training time, memory, and parameter overhead.CSRNet outputs density maps at 1/8 of the input size and uses bilinear resolution as the input image.
- Ground-truth generation: Ground-truth density maps blur head annotations with normalized geometry-adaptive Gaussian kernels whose σ_i depends on nearest-neighbor distances.The experiments use β = 0.3 and k = 3; sparse crowds instead use the average head size for blurring.
- Training details: Training uses cropped and mirrored patches, VGG-16 fine-tuning for the first 10 convolutional layers, Gaussian initialization elsewhere, SGD, and Euclidean distance.The training learning rate is fixed at 1e-6, and the loss compares estimated and ground-truth density maps.
4. Experiments
CSRNet is evaluated across crowd-counting and vehicle-counting datasets using error, density-map quality, and ablation analyses. The experiments report strong performance across varied scenes, with dilated configurations and ROI-aware evaluation supporting the comparisons.
- Density-map evaluation: PSNR and SSIM are used to assess generated density-map quality after resizing and normalizing predicted and ground-truth maps.The supplementary material presents density-map samples across five datasets and varied density levels.
- Ablation study: CSRNet B achieves the lowest error among four equal-parameter configurations on ShanghaiTech Part A and is selected for subsequent experiments.The configurations differ in their back-end dilation rates; dropout provides no significant improvement.
- ShanghaiTech dataset: 7% lower MAE than CP-CNN is achieved on ShanghaiTech Part A, while Part B shows 47.3% lower MAE than CP-CNN.CSRNet also achieves the highest SSIM and PSNR for density maps on ShanghaiTech Part A.
- Other crowd datasets: The evaluation covers datasets with varied crowd densities, perspectives, resolutions, and surveillance settings, including UCF CC 50 and UCSD.The supplied passages identify their estimation-error tables but do not report the corresponding numerical outcomes.
- WorldExpo’10 dataset: CSRNet achieves the best average accuracy and the best accuracy in 4 of 5 WorldExpo’10 scenes.The dataset uses ROI-masked frames and dot maps during preprocessing.
- TRANCOS vehicle counting: 67.7%, 60.1%, 48.7%, and 22.2% lower GAME(0), GAME(1), GAME(2), and GAME(3), respectively, are reported against the method in on TRANCOS.GAME(L) sums regional MAE over a grid of 4^L non-overlapping regions; at L = 0 it equals MAE.
5. Conclusion
The conclusion presents CSRNet as an easy-trained architecture for crowd counting and high-quality density-map generation. It attributes the design’s contextual aggregation to dilated convolutions and reports state-of-the-art crowd-counting and vehicle-counting performance.
- CSRNet is proposed as an easy-trained end-to-end architecture for crowd counting and high-quality density-map generation.
- Dilated convolutional layers aggregate multi-scale contextual information while expanding the receptive field without losing resolution.
- The model achieves state-of-the-art performance on four crowd-counting datasets and best accuracy on the vehicle-counting task.
7. Appendix: supplementary material
The appendix supplies additional CSRNet outputs from five datasets to demonstrate the design’s validity. It evaluates density-map quality with PSNR and SSIM across varied density levels.
- Additional CSRNet results are presented for ShanghaiTech, UCF CC 50, WorldExpo’10, UCSD, and TRANCOS.
- PSNR and SSIM are used to evaluate the quality of generated density maps.
- The supplementary samples represent a variety of density levels across the five datasets.
- Figures 7 through 12 show original or ROI-masked images, ground-truth density maps, and CSRNet-generated density maps.