Source-linked AI summary
Iterative Crowd Counting
Viresh Ranjan, Hieu Le, Minh Hoai
TL;DR
Crowd counting requires accurate estimation across images with widely varying densities, while manual counting is slow and unreliable. The paper proposes ic-CNN, which predicts a low-resolution density map before refining it with a second branch, and extends this design to multiple stages. The method achieves state-of-the-art performance across three challenging benchmarks, including a 48.3% MAE improvement on Shanghaitech Part B over previously published results.
Problem
Crowd counting must estimate people in scenes ranging from a few to thousands, but manual counting is slow and unreliable and a single CNN struggles with the density variation.
Method
ic-CNN uses a low-resolution CNN branch whose prediction and convolutional features feed a high-resolution branch, with a multi-stage extension combining successive predictions.
Results
ic-CNN achieves state-of-the-art results on multiple datasets, including a 48.3% MAE improvement over previously published Shanghaitech Part B results and the best MAE on UCF.
Takeaways & Limitations
The experiments support iterative density-map prediction as an effective approach across Shanghaitech, WorldExpo'10, and UCF crowd-counting benchmarks.
Takeaways & Limitations
On qualitative Shanghaitech Part-A results, ic-CNN sometimes misclassifies tree leaves as tiny people in a crowd.
Abstract
from arXiv · showhide
In this work, we tackle the problem of crowd counting in images. We present a Convolutional Neural Network (CNN) based density estimation approach to solve this problem. Predicting a high resolution density map in one go is a challenging task. Hence, we present a two branch CNN architecture for generating high resolution density maps, where the first branch generates a low resolution density map, and the second branch incorporates the low resolution prediction and feature maps from the first branch to generate a high resolution density map. We also propose a multi-stage extension of our approach where each stage in the pipeline utilizes the predictions from all the previous stages. Empirical comparison with the previous state-of-the-art crowd counting methods shows that our method achieves the lowest mean absolute error on three challenging crowd counting benchmarks: Shanghaitech, WorldExpo'10, and UCF datasets.
1 Introduction
Crowd counting is important but difficult because crowd densities vary widely, motivating density-estimation CNNs and the proposed iterative two-branch architecture. ic-CNN combines low-resolution predictions with convolutional features to produce high-resolution density maps, with a multi-stage extension for further refinement.
- Motivation: Manual crowd counting is difficult, laborious, slow, and unreliable, motivating automatic image- and video-based counting.The need spans applications including journalism and public safety.
- Challenge: Crowd density can range from a few people to a few thousand, making the full spectrum difficult for a single CNN to handle.Earlier methods addressed this variation with multi-column or switching CNN architectures using different receptive field sizes.
- Existing approach: Density-estimation CNNs predict a density value for each pixel, whose sum yields the final crowd count.Unlike detection-then-counting, density estimation need not make early binarized decisions.
- Proposed method: ic-CNN uses two branches: LR-CNN predicts a low-resolution density map, while HR-CNN combines it with convolutional features to predict a full-resolution map.The low-resolution map is produced at 1/4 the original image size and passed to the second branch with convolutional features.
- Proposed method: The multi-stage ic-CNN extension combines multiple ic-CNN models sequentially, with later stages using predictions from earlier stages.This design is intended to further improve predicted density-map quality.
2 Related Work
Earlier crowd-counting research progressed from detection and regression approaches toward CNN-based density estimation. Related methods addressed density variation using multi-column, switching, and context-estimator architectures, while also exposing the value of density annotation maps.
- Detection-based methods: Early crowd-counting methods framed the task as object detection, but often failed to count people accurately in extremely dense scenes.Examples used Haar or HOG-based head-related features and detectors.
- CNN-based alternatives: CNN-based regression and density-category classification methods predicted counts or categories directly but overlooked crowd density annotation maps.These methods included direct image-to-count regression and a cascade of CNNs trained on misclassified images.
- Multi-column CNNs: MCNN addressed variation in crowd density and scale with three columns using different filter and receptive-field sizes.Patch-based training was used to prevent overfitting and improve performance through data augmentation.
- Switching CNNs: Switching CNNs decoupled multi-column branches so each CNN could focus on a density sub-range rather than fusing all columns for every patch.The related methods built on specialization by density and receptive-field scale.
3 Proposed Approach
ic-CNN uses a two-branch architecture to predict high-resolution crowd density maps from a low-resolution prediction and convolutional features. A multi-stage extension sequentially combines predictions from earlier blocks, with stage-wise training and frozen preceding parameters.
- 3.1 Iterative Counting CNN: ic-CNN maps each input image to a low-resolution density prediction, then uses it with low-resolution features to produce a full-resolution density map.The high-resolution branch predicts a map at the same size as the original image.
- 3.1 Iterative Counting CNN: The low-resolution prediction contains spatial crowd-distribution information and is included as an important feature for high-resolution prediction.
- 3.1 Iterative Counting CNN: The branches are trained jointly with a weighted loss combining low- and high-resolution errors, using squared error as a reasonable loss choice.The scalar weights λl and λh control the relative importance of the two loss terms.
- 3.1 Iterative Counting CNN: At test time, the high-resolution HR-CNN output is used as the final density map, and the crowd count is obtained by summing its pixels.
- 3.2 Multi-stage Crowd Counting: A multi-stage ic-CNN combines multiple ic-CNN blocks, with each block using the image, its low-resolution features and prediction, and earlier blocks’ low- and high-resolution maps.Earlier block parameters are frozen before training the next stage.
- 3.3 Training Details: 48.3% improvement in MAE and 46.8% improvement in RMSE were reported on Shanghaitech Part B versus previously reported state-of-the-art results using one-stage ic-CNN.The one-stage model uses two branches, while the comparison baseline is CP-CNN’s five-branch architecture.
4 Experiments
Experiments evaluate ic-CNN on three challenging crowd-counting datasets using MAE and RMSE, with comparisons, ablations, efficiency measurements, and density-stratified analysis. The method generally outperforms prior approaches, while intermediate-resolution, feature sharing, and low-resolution feedback materially affect performance.
- Shanghaitech results: On Shanghaitech, ic-CNN outperforms previous approaches in three of four cases; Part-B improves MAE by 48.3% and RMSE by 46.8% over prior state-of-the-art results.On Part A, ic-CNN improves MAE over CP-CNN by 5.1 absolute points, while two stages improve MAE by 1.3 over one stage.
- Shanghaitech ablations: Using any intermediate resolution other than 1/4 causes a performance drop, while the high-resolution branch remains fixed at the input-image resolution.The resolution study varies the low-resolution branch while keeping the high-resolution branch at the original image size.
- Efficiency: ic-CNN trains in 10 hours and has significantly fewer parameters than Switching CNN and CP-CNN, although comparison training times for MCNN and CP-CNN were unavailable.Training was performed on a single Nvidia GTX 1080 TI GPU.
- Shanghaitech ablations: Feature sharing and low-resolution prediction feedback are both important: removing either component causes a significant performance drop.The full ic-CNN shares both convolutional features and the low-resolution prediction map with HR-CNN.
- Other dataset results: On WorldExpo’10, ic-CNN outperforms previous approaches in three of five cases, while on UCF it achieves the best MAE and surpasses CP-CNN by a large margin.The WorldExpo’10 table caption reports three wins out of six cases, whereas the accompanying experimental text reports three out of five cases.
5 Conclusions
The paper concludes that ic-CNN provides a two-branch density-estimation architecture and extends it into a multi-stage pipeline, with experiments demonstrating effectiveness on challenging benchmarks.
- 5 Conclusions: ic-CNN uses a two-branch architecture for crowd counting through density estimation.The paper also proposes a multi-stage pipeline in which each stage uses predictions from all previous stages.
- 5 Conclusions: The multi-stage pipeline combines multiple ic-CNNs sequentially, with each stage incorporating earlier predictions.
- 5 Conclusions: Experiments on three challenging crowd-counting benchmark datasets demonstrate the effectiveness of the iterative approach.
- 5 Conclusions: Qualitative results include success and failure cases, with some failures caused by misclassifying tree leaves as people.The figures compare input images, ground-truth maps, low-resolution predictions, and high-resolution predictions.
- 5 Conclusions: Shanghaitech Part B qualitative results report rounded total counts beneath the density maps.