PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimation

Vishwanath A. Sindagi, Vishal M. Patel

arXiv:1707.01202v1cs.CV

TL;DR

Crowd counting and density estimation are important but difficult because crowded images contain occlusion, clutter, and scale and perspective variation. The paper surveys traditional and recent CNN-based single-image methods, datasets, strengths, drawbacks, and research directions; it reports that CNN-based methods generally outperform traditional approaches, especially in dense and varied scenes, while scale- and context-aware models drive much of the improvement.

  • Problem

    Crowd counting and density estimation must handle occlusions, clutter, non-uniform density, and scale and perspective variation across crowded scenes.

  • Method

    The paper surveys traditional hand-crafted approaches and recent CNN-based single-image methods, reviews challenging datasets, and discusses model merits, drawbacks, and future directions.

  • Results

    CNN-based methods generally outperform traditional approaches across datasets and are more effective in large-density crowds with diverse scene conditions.

  • Takeaways & Limitations

    The survey concludes that CNN-based approaches are more adept at handling large-density crowds with variations in object scales and scene perspective.

  • Takeaways & Limitations

    Scale invariance in multi-network methods remains limited by the number of columns and receptive-field sizes selected for dataset scales, while global context is not explicitly modeled.

Abstract

from arXiv · show

Estimating count and density maps from crowd images has a wide range of applications such as video surveillance, traffic monitoring, public safety and urban planning. In addition, techniques developed for crowd counting can be applied to related tasks in other fields of study such as cell microscopy, vehicle counting and environmental survey. The task of crowd counting and density map estimation is riddled with many challenges such as occlusions, non-uniform density, intra-scene and inter-scene variations in scale and perspective. Nevertheless, over the last few years, crowd count analysis has evolved from earlier methods that are often limited to small variations in crowd density and scales to the current state-of-the-art methods that have developed the ability to perform successfully on a wide range of scenarios. The success of crowd counting methods in the recent years can be largely attributed to deep learning and publications of challenging datasets. In this paper, we provide a comprehensive survey of recent Convolutional Neural Network (CNN) based approaches that have demonstrated significant improvements over earlier methods that rely largely on hand-crafted representations. First, we briefly review the pioneering methods that use hand-crafted representations and then we delve in detail into the deep learning-based approaches and recently published datasets. Furthermore, we discuss the merits and drawbacks of existing CNN-based approaches and identify promising avenues of research in this rapidly evolving field.

1. Introduction

Crowd counting and density estimation support monitoring and scene understanding but remain difficult because crowded scenes exhibit occlusion, clutter, and major variation in appearance, scale, and perspective. This survey addresses the gap in earlier surveys by reviewing recent CNN-based single-image methods and challenging datasets.

  • 1. Introduction: Crowd counting measures the number of people, while density estimation maps an image to per-pixel people density; researchers often address them jointly.These tasks support higher-level capabilities including crowd monitoring and scene understanding.
  • 1. Introduction: Crowd analysis is important for monitoring, management, safety, and security in increasingly frequent gatherings such as sporting events and public demonstrations.Applications include surveillance in stadiums, tourist spots, shopping malls, and airports.
  • 1. Introduction: Occlusions, high clutter, non-uniform density and illumination, and intra-scene and inter-scene variation in appearance, scale, and perspective make crowd analysis difficult.These challenges are illustrated across parades, concerts, demonstrations, and sports stadiums.
  • 1. Introduction: Crowd counting and density estimation are fundamental tasks that form building blocks for related applications including segmentation, behavior analysis, tracking, scene understanding, and anomaly detection.Their applications extend across computer vision, public safety, and other disciplines.
  • 1. Introduction: Earlier surveys largely covered hand-crafted traditional methods, whereas this paper surveys recent CNN-based approaches and newly created challenging datasets.The survey also discusses CNN-based methods’ merits and drawbacks and identifies future research directions.

2. Review of traditional approaches

Traditional crowd counting methods include detection-, regression-, and density-estimation-based approaches. Regression methods reduce reliance on detectors, while later density-map methods incorporate spatial information and address nonlinear mappings, computational cost, and feature richness.

  • 2. Review of traditional approaches: Traditional crowd counting methods are broadly categorized as detection-based, regression-based, and density estimation-based approaches.The survey briefly reviews detection and regression methods using hand-crafted features.
  • 2.1. Detection-based approaches: Detection-based methods use sliding-window detectors to identify people and then count them, but parts-based and shape-based detectors fail in extremely dense, cluttered crowds.Detection may follow monolithic or parts-based frameworks.
  • 2.2. Regression-based approaches: Regression-based methods learn mappings from local image-patch features to counts, avoiding dependence on learning complex detectors.They combine low-level feature extraction with regression models such as linear, ridge, Gaussian-process, and neural-network regression.
  • 2.2. Regression-based approaches: High-density counting is hindered by low resolution, severe occlusion, foreshortening, perspective, and the unreliability of any single feature or detection method.Spatial relationships between neighboring regions can constrain count estimates.
  • 2.3. Density estimation-based approaches: Later methods address nonlinear feature-to-density mappings, sparse and imbalanced data, computational complexity, and limited feature sets using random forests, cumulative attributes, subspace learning, and richer features.A crowdedness prior and separate forests were used to handle differences between crowded and non-crowded patches.
  • 2.3. Density estimation-based approaches: Density-map regression incorporates spatial information by mapping local patch features to object density maps instead of regressing only global counts.This avoids the difficult task of detecting and localizing individual object instances.

3. CNN-based methods

CNN-based crowd counting methods are categorized by network properties and inference methodology, spanning basic, scale-aware, context-aware, patch-based, and whole-image approaches. The survey reviews representative architectures that address scene transfer, training efficiency, global context, scale variation, and related counting tasks.

  • Network-property categorization: CNN-based approaches are categorized by network properties and training approach, including basic, scale-aware, and context-aware models.Scale-aware models use techniques such as multi-column or multi-resolution architectures to handle scale variation.
  • Inference-method categorization: Inference is divided into patch-based methods that aggregate sliding-window predictions and whole-image methods that avoid computationally expensive sliding windows.Whole-image inference processes the complete image directly rather than repeatedly evaluating cropped patches.
  • Representative CNN-based methods: Cross-scene counting adapts image-to-count mappings to new target scenes by alternatively training on crowd-count and density-estimation objectives.The associated density-map generation incorporates perspective information for perspective normalization and robustness to scale and perspective variations.
  • Representative CNN-based methods: Layered boosting adds CNN layers iteratively to estimate residual errors, while selective sampling reduces the influence of trivial or outlier training samples.Each newly added layer is trained on the difference between the prior estimate and ground truth.
  • Representative CNN-based methods: End-to-end counting uses entire images, shared computation across overlapping regions, local-count estimation, GoogLeNet features, and LSTM decoders.This design incorporates contextual information while reducing computational complexity relative to patch-based processing.
  • Representative CNN-based methods: Multi-column and deep–shallow architectures address scale and perspective variation, while CNNs also support counting cars, penguins, and line-crossing people.The reviewed examples include MCNN, a deep–shallow density model, and a two-task density-and-velocity model for crossing-line counting.

4. Discussion

The discussion contrasts patch-based, scale-aware, and multi-task CNN approaches, emphasizing trade-offs involving global context, inference efficiency, scale variation, and dataset dependence.

  • Patch-based approaches: Patch-based methods can lose global context and incur inefficient sliding-window evaluation, while perspective-map requirements may limit applicability.These methods may also ignore scale variance across datasets by assuming it will be learned implicitly.
  • Scale-aware approaches: Multi-network scale-aware methods improve performance, but their scale invariance depends on dataset-specific columns and receptive-field sizes.They also do not explicitly model global context, which is important for crowd counting.
  • Scale-aware approaches: Multi-scale averaging offers a simple way to address scale variation, but it makes inference inefficient and does not explicitly encode global image context.The discussion presents local-and-global-context modeling as a separate response to this limitation.
  • Multi-task approaches: Multi-task methods can improve performance by learning high-level priors, but the number of density classes must be selected according to each dataset.The density-level classification design is therefore dataset dependent.

5. Datasets and results

The survey reviews datasets spanning low- and high-density scenes, varied perspectives, and cross-scene settings, then compares traditional and CNN-based counting results. CNN methods generally perform better, particularly when models incorporate scale and context information, although dataset limitations remain important.

  • Datasets: UCF CC 50 contains 50 diverse images spanning 94 to 4543 people, but its limited image count requires five-fold cross-validation and challenges CNN evaluation.The dataset has 63,075 labelled individuals and an average of 1280 individuals per image.
  • Datasets: WorldExpo ’10 evaluates cross-scene counting across 103 training scenes and five test scenes, but its test diversity and maximum count are limited.The test crowd count ranges from 1 to 220, limiting evaluation for extremely dense crowds across varied scenes.
  • Datasets: ShanghaiTech contains 1198 images and 330,165 annotated heads across Internet-sourced Part A and metropolitan-street Part B.Part A contains considerably denser images than Part B.
  • Datasets: UCSD and Mall emphasize relatively low-density, single-scene imagery, whereas later datasets provide broader density and perspective variation.Sample images and the dataset summary illustrate this contrast across UCSD, Mall, UCF CC 50, WorldExpo ’10, and ShanghaiTech.
  • Results: CNN-based methods generally outperform traditional approaches across the reviewed crowd-counting datasets.The comparison uses standard count-estimation error metrics including MAE and MSE.
  • Results: Scale-aware and context-aware CNN models account for most performance improvements, with higher model complexity associated with lower count error.The survey attributes this pattern to adding scale and context information.
  • Results: Accurate crowd counts do not necessarily produce high-quality density maps, as illustrated by Zhang et al.’s ShanghaiTech results.The reported estimated density maps are described as poor despite accurate crowd-count estimation.

6. Future research directions

The survey identifies dataset imbalance, costly scene-specific annotation, poor density-map quality, and limited real-time integration as major directions for future work. It also anticipates deeper, scale-aware, and context-aware CNN architectures as important avenues.

  • Future directions: Deeper CNN architectures are expected to dominate future crowd-counting and density-estimation research.This forecast is based on analyses of existing methods and results alongside broader computer-vision developments.
  • Future directions: Large-scale datasets for extremely dense crowds remain essential, because existing coverage is limited and ShanghaiTech has non-uniform density-level sampling.UCF CC 50 targets large-density crowds but is too small for training deeper networks; ShanghaiTech contains many more low-density than high-density samples.
  • Future directions: Transfer learning and domain adaptation deserve further study because retraining models for every new scene makes annotation costly and impractical.Existing methods commonly retrain on new scenes, while transfer learning for crowd scenes remains relatively unexplored.
  • Future directions: Real-time video counting remains challenging because combining image- and video-based techniques is non-trivial and requires low-latency methods.
  • Future directions: Future methods should generate high-quality density maps while preserving low count-estimation error.Euclidean-loss regression on down-sampled maps can yield accurate counts but poor maps, which can adversely affect downstream cognition tasks.
  • Future directions: Incorporating additional contextual and scale information is identified as a route to further progress in crowd counting and density estimation.

7. Conclusion

The article surveys traditional and CNN-based crowd-counting and density-estimation methods, organizing CNN approaches by training process and network properties. It concludes that CNN methods better handle dense, scale-varying, perspective-varying crowds, while scale and contextual information substantially improve estimation error.

  • Conclusion: The survey categorizes crowd-counting methods into traditional hand-crafted approaches and CNN-based approaches, further grouping CNN methods by training process and network property.
  • Conclusion: CNN-based methods are more adept at handling large-density crowds with variations in object scale and scene perspective.
  • Conclusion: Incorporating scale and contextual information drastically improves estimation error in CNN-based methods.
  • Conclusion: The survey reviews a representative subset of recent literature and identifies compelling challenges for future crowd-counting and density-estimation research.
Loading 1707.01202v1…