PaperlayerView exact revision on arXiv

Source-linked AI summary

CNN-based Density Estimation and Crowd Counting: A Survey

Guangshuai Gao, Junyu Gao, Qingjie Liu, Qi Wang, Yunhong Wang

arXiv:2003.12783v1cs.CV

TL;DR

Crowd-counting research has produced many methods, but their effectiveness and density-map quality require systematic study. This paper surveys CNN-based models, benchmarks representative approaches, and analyzes leading performers to derive future directions and connections to object counting.

  • Problem

    Most existing methods emphasize the count rather than density-map quality, although density-map quality affects performance.

  • Method

    The paper surveys CNN-based crowd-counting models by learning paradigm, inference manner, supervision form, generalization scope, and supervision level, then benchmarks representative methods.

  • Results

    CNN-based methods substantially outperform traditional approaches across six mainstream crowd-counting benchmark datasets.

  • Takeaways & Limitations

    The survey aims to provide insights for future crowd-counting research and feasible solutions for object counting in other domains.

Abstract

from arXiv · show

Accurately estimating the number of objects in a single image is a challenging yet meaningful task and has been applied in many applications such as urban planning and public safety. In the various object counting tasks, crowd counting is particularly prominent due to its specific significance to social security and development. Fortunately, the development of the techniques for crowd counting can be generalized to other related fields such as vehicle counting and environment survey, if without taking their characteristics into account. Therefore, many researchers are devoting to crowd counting, and many excellent works of literature and works have spurted out. In these works, they are must be helpful for the development of crowd counting. However, the question we should consider is why they are effective for this task. Limited by the cost of time and energy, we cannot analyze all the algorithms. In this paper, we have surveyed over 220 works to comprehensively and systematically study the crowd counting models, mainly CNN-based density map estimation methods. Finally, according to the evaluation metrics, we select the top three performers on their crowd counting datasets and analyze their merits and drawbacks. Through our analysis, we expect to make reasonable inference and prediction for the future development of crowd counting, and meanwhile, it can also provide feasible solutions for the problem of object counting in other fields. We provide the density maps and prediction results of some mainstream algorithm in the validation set of NWPU dataset for comparison and testing. Meanwhile, density map generation and evaluation tools are also provided. All the codes and evaluation results are made publicly available at https://github.com/gaoguangshuai/survey-for-crowd-counting.

I. INTRODUCTION

Crowd counting matters for social safety and has developed from detection and regression methods toward CNN-based density estimation. This survey focuses on systematically organizing and evaluating those modern approaches while considering their relevance to other object-counting domains.

  • Crowd-counting techniques can potentially generalize to related domains including cell microscopy, animals, vehicles, leaves, and environmental surveys.
  • Crowd counting supports social safety and control management in rapidly growing gatherings such as parades, concerts, and stadiums.
  • Existing approaches span detection-based, regression-based, density-estimation, and CNN-based density-estimation methods.
  • Detection methods struggle with occlusion and background clutter in extremely dense crowds, while regression methods generally ignore spatial information.
  • CNN-based methods improved density estimation through stronger feature representation, with fully convolutional networks becoming the mainstream architecture.
  • The survey emphasizes modern CNN-based density-estimation methods, classifying them by architecture, supervision, learning paradigm, cross-scene or multi-domain influence, and other factors.

C. Contributions of this paper

The paper contributes a systematic, multi-perspective survey of CNN-based crowd-counting research, supported by taxonomies, chronology, structural comparisons, and discussion of future directions. It also analyzes challenge factors and distinguishes its review from earlier surveys.

  • Contributions: The paper provides a comprehensive and systematic overview of CNN-based models organized by network architecture, supervised form, and learning paradigm.
  • Contributions: It analyzes state-of-the-art performance by model attributes and techniques, while discussing challenge factors that motivate more effective algorithms.
  • Contributions: The survey compares existing density-map network structures and presents a chronology spanning traditional heuristic methods, CNN-based models, and emerging localization or tracking directions.
  • Contributions: The paper examines open questions in model design, dataset collection, and generalization to other domains through domain adaptation or transfer learning.
  • Contributions: Its detailed and in-depth review is presented as substantially different from previous review and survey works.
  • Organization: The paper organizes its literature review, datasets, metrics, benchmarking, discussion, and future directions across the remaining sections.

A. Representative network architectures for crowd counting

Crowd-counting architectures evolved from simple CNNs to multi-column and single-column designs that address scale and contextual variation, while multi-column models incur substantial complexity and redundancy.

  • 1) Basic CNN: Basic CNN methods use standard convolution, pooling, and fully connected layers, making them simple to implement but typically less accurate.Early models used cascaded classifiers, expanded negative samples, or layer-wise boosting to improve counting.
  • 2) Multi-column: Multi-column networks use branches with different receptive fields to capture multi-scale crowd information.MCNN uses large, medium, and small kernels, while other models combine shallow, deep, contextual, or attention-based branches.
  • 2) Multi-column: Multi-column architectures are difficult to train and often contain bloated, redundant structures.Their density-level classifiers and parallel branches increase training time, parameters, and architectural complexity.
  • 3) Single column: These limitations motivate simpler single-column networks that use deeper CNNs without proportionally increasing network complexity.The survey describes single-column models using dilated convolutions, scale adaptation, multi-scale encoders, and enhanced feature representations.

3) Single column:

Single-column architectures replace bloated multi-branch designs with deeper networks and specialized feature modules, while crowd-counting systems also vary by learning paradigm, inference strategy, and task combination.

  • 3) Single column: Single-column networks use one deeper CNN architecture rather than the bloated structure of multi-column networks.Examples include scale adaptation, dilated convolutions, Inception-based encoders, and transposed-convolution decoders.
  • 3) Single column: TEDnet uses multiple decoding paths and dense skip connections to capture multi-scale features and improve supervised information flow.Its combinational loss includes local coherence and spatial correlation terms to address gradient vanishing and back-propagation.
  • 3) Single column: Single-column architectures have attracted increasing attention because of their architectural simplicity and training efficiency.This trend is presented as a response to the complexity and inefficiency of multi-column designs.
  • Learning paradigm: Crowd-counting networks are classified by learning paradigm into single-task and multi-task methods.Single-task methods usually estimate density maps and sum their pixels, whereas multi-task methods combine counting with related tasks.
  • Learning paradigm: Multi-task methods combine density estimation with classification, detection, or segmentation through multiple task-specific subnetworks or branches.CMTL uses count classification as a high-level prior, while Decidenet adaptively selects detection- or regression-based density estimation.
  • Inference manner: CNN-based approaches also differ in inference manner, with patch-based methods using cropped windows and whole-image methods processing the complete image.Patch-based inference assembles sliding-window estimates, while whole-image inference can preserve global information but may lose local detail.

2) Whole image-based methods:

Whole-image inference processes an entire image rather than sliding patches, preserving global context but potentially losing local information; related supervision and domain strategies address labeling and generalization challenges.

  • 2) Whole image-based methods: Whole-image methods take the complete image as input and output a density map or total crowd count.Compared with patch-based inference, they avoid sliding-window computation and preserve global information, but may lose local information.
  • Supervision form: Crowd-counting methods are categorized by annotation use into fully supervised and un-, self-, or semi-supervised approaches.The distinction concerns whether human-labeled annotations are used during training.
  • Supervision form: Large-scale hand-annotated data create substantial labeling costs and can still lead to overfitting when models transfer across domains.These constraints motivate training with fewer or no labeled annotations.
  • Supervision form: Un-, self-, and semi-supervised methods use few or no ground-truth labels, auxiliary related tasks, or unlabeled data.Some reported methods achieve performance comparable to supervised approaches, including unsupervised feature learning and semi-supervised GAN-based regression.
  • Cross-domain counting: Domain adaptation is presented as a tool for building counting models that generalize beyond a specific object domain.Examples include matching-based counting, domain-specific normalization layers, and synthetic-to-real adaptation using CycleGAN variants.
  • Supervision level: Image-level supervision estimates object counts without requiring instance locations, unlike instance-level supervision based on points or bounding boxes.ILC jointly estimates category density maps, total object counts, and spatial distributions.

1) Instance-level supervision:

Instance-level crowd-counting research relies on point or bounding-box annotations and is evaluated across datasets designed to represent varied scenes, densities, viewpoints, and environmental conditions.

  • 1) Instance-level supervision: Instance-level supervision labels each object location with a point or bounding box, and most crowd-density methods use this form of annotation.Such labeling requires hand-annotated instance locations.
  • Datasets: Crowd-counting datasets have expanded to represent scale variation, clutter, changing environments, illumination differences, and other practical challenges.The survey reviews real-world and synthetic datasets and summarizes representative benchmarks.
  • A. Most frequently-used datasets: UCSD is an early sidewalk-camera dataset with 2,000 frames at 238×158 resolution and annotations for each pedestrian every five frames.Labels for intervening frames are created by linear interpolation, and the single location provides no perspective change across frames.
  • A. Most frequently-used datasets: Mall contains 2,000 surveillance frames and 62,325 pedestrians, with varied densities, activities, illumination, perspective distortion, and occlusion.Its conditions provide greater scene and appearance variation than UCSD.
  • A. Most frequently-used datasets: WorldExpo’10 contains 1,132 annotated video sequences from 108 surveillance cameras and 3,920 frames at 576×720 resolution.It is described as a large data-driven cross-scene dataset collected during the 2010 Shanghai World Expo.
  • B. More recently datasets: UCF-QNRF contains 1,535 challenging images and about 1.25 million annotations spanning diverse viewpoints, densities, scenes, and lighting conditions.Some high-resolution images may cause GPU memory issues when training on entire scenes.
  • B. More recently datasets: GCC contains 15,212 synthetic GTA5 images with 7,625,843 persons, while NWPU-Crowd contains 5,109 images and 2,133,238 annotated instances.GCC is highlighted for free annotation, large volume, high resolution, diverse scenes, and accurate annotations.
  • Specialized datasets: Specialized datasets target settings such as line counting, indoor scenes, train and subway stations, bridges, airports, and multi-source surveillance.These datasets address scenario-specific crowd-counting requirements.

D. Representing object Counting datasets in other fields

The survey presents object-counting datasets beyond crowd counting to assess model generalization, and reviews metrics for image-level accuracy, localization, and density-map quality.

  • TRANCOS evaluates vehicle counting in traffic-jam images and is used to assess crowd-counting models’ generalization ability.
  • CARPK contains nearly 90,000 cars from four parking lots, with drone-view images and bounding-box annotations.
  • Evaluation metrics cover image-level counting performance, pixel-level density-map quality, and point-level localization precision.
  • MAE measures estimate accuracy, whereas RMSE indicates robustness; GAME and patch metrics further evaluate regional errors, while MPAE measures density localization errors.

B. Pixel-level metrics

Pixel-level metrics assess generated density-map quality, while localization metrics evaluate how accurately predicted objects are positioned.

  • Pixel-level metrics: PSNR evaluates density-map quality through corresponding-pixel error, with higher values indicating smaller errors.
  • Pixel-level metrics: SSIM measures brightness, contrast, and structure similarity on a [0,1] scale, where larger values indicate less distortion.
  • Localization metrics: Average Precision and Average Recall are common metrics for localization performance, requiring a trade-off because increasing AP generally decreases AR.
  • Benchmark context: The benchmark compares 53 CNN-based methods and 7 traditional approaches across six mainstream crowd-counting datasets using MAE and RMSE.
  • Benchmark findings: CNN-based methods substantially outperform traditional models, while performance has improved progressively since the first CNN-based density-estimation model in 2015.
  • Benchmark findings: Cross-scene models perform worse than single-scene and domain-specific models but provide a solution for generalizing trained models to unseen scenes.

B. Properties-based evaluation

The survey relates model properties to crowd-counting performance, emphasizing architectures and mechanisms for handling scale, rotation, density, perspective, and background variation.

  • Architecture: Among selected top-performing models, two-thirds use single-column architectures, suggesting deeper networks may be preferable to wider ones.
  • Feature mechanisms: Visual attention highlights crowd-containing regions and filters background clutter, while dilated convolution and spatial pyramid pooling improve feature processing.
  • Geometric variation: Spatial transformer networks and deformable convolutions address rotation, scaling, and warping that limit standard CNN feature invariance.
  • Perspective and scale: Perspective information provides additional support for extracting multi-scale features in crowd-counting networks.
  • Density variation: Pan-density learning combines global features with local estimation to address varying scene densities and inconsistent local densities.
  • Benchmark interpretation: Table IV compares methods using dataset-specific MAE and RMSE results, with colors marking first-, second-, and third-best performance.
  • Training design: Multi-pathway or multi-task frameworks with joint loss functions can improve estimation performance and accelerate training.

C. Attributes-based analysis

CNN-based crowd counting has advanced, but robust performance remains constrained by varied scene conditions and design choices spanning density-map generation, losses, cue fusion, and network topology.

  • Challenges: Crowd-counting models must handle occlusion, scale variation, perspective distortion, rotation, illumination, weather, and mixed indoor, outdoor, and in-the-wild scenes.These attributes can occur together, increasing the difficulty of designing robust networks.
  • Ground truth density maps generation: High-fidelity ground-truth density maps are essential, with Gaussian-kernel strategies extended using perspective information to address scale variation.Perspective-aware generation introduces the additional requirement of obtaining a perspective map.
  • Ground truth density maps generation: Content-aware alternatives include nearest-neighbor techniques, inverse k-nearest-neighbor maps, and learnable density-map representations from dot labels.These approaches target smoother training, accurate localization, or adaptive density-map representations.
  • Loss function: Customized losses address spatial density variation, including Maximum Excess over Pixels loss for high-discrepancy pixel-level subregions.The survey identifies loss design as an essential training procedure for effective models.
  • Information fusion of multi cues: Fusing scale-aware, context-aware, sparse-scene, dense-scene, geometric, semantic, and numeric cues can improve crowd-counting performance.The survey presents heterogeneous-cue fusion as a promising direction for broader information integration.

B. Dataset construction

Crowd-counting datasets must represent diverse scenes, views, and annotations while supporting reliable density-map evaluation and cross-domain generalization.

  • Scene diversity: Single-scene datasets lack perspective variation, motivating cross-scene datasets such as UCF CC 50, ShanghaiTech Part A, and UCF QNRF.The survey notes limitations including scarce high-resolution images and intra-class variation that can exceed network capability.
  • Scene diversity: Synthetic GCC data provides diverse scenes for wild conditions, but its domain gap from real data limits direct equivalence.The domain gap is also associated with poorer density-map quality for SE Cycle GAN on the SHT A dataset.
  • Multi-view: Single-view datasets cannot fully capture wide scenes with distant, low-resolution, or heavily occluded crowds, motivating multi-view collection.City street data is cited as containing broad crowds with complex occlusion and scale variation.
  • Annotation accuracy: Cross-scene, multi-view, accurately annotated datasets are presented as essential for reflecting real-world challenges and improving generalization.Effective annotation tools are also identified as vital for dataset construction.
  • Density-map quality: Most methods prioritize count accuracy over density-map quality, although density-map quality affects performance and can be assessed with PSNR and SSIM.The survey reports SE Cycle GAN as the worst performer in the cited SHT A comparison.
  • Cross-dataset evaluation: Cross-dataset evaluation compares oracle training on Shanghai Tech Part A with training on NWPU-Crowd and testing on Shanghai Tech Part A.The cited table reports MAE/RMSE and corresponding performance degradations.

E. Robustness for background

Robust crowd counting requires rejecting background distractors, extending beyond task-specific models, and balancing lightweight efficiency against accuracy.

  • Background robustness: JHU-CROWD and NWPU-Crowd introduce distractors and negative samples without person or crowd objects to test zero-density background responses.JHU-CROWD contains 100 distractors, while NWPU-Crowd contains 351 negative samples.
  • Background robustness: Current models mistakenly estimate density on distractors and negative samples, while PCC-Net benefits from segmentation information separating foreground heads from background.PCC-Net outperforms several VGG-backbone methods among the compared light and VGG-based models.
  • Universality or generalization: Multi-task learning using counting, segmentation, or group detection is suggested for extracting large-range features.A unified principle and framework across counting tasks is described as promising but difficult.
  • Universality or generalization: A universal object-counting model remains a meaningful challenge and a stringent test of robustness and generalization beyond task-specific designs.The survey identifies commonalities across crowd, vehicle, and cell counting as motivation for universality.
  • Lightweight network: Current CNN models often have millions of parameters and high FLOPs, while pretraining on classification datasets may affect regression performance and requires substantial time.These constraints motivate lightweight architectures that reduce computation without sacrificing accuracy.
  • Lightweight network: 2138 × fewer parameters distinguish LCNN from CP-CNN in the cited comparison, with the survey associating LCNN’s compactness with shallow, non-pretrained design.The survey uses parameter count as an indicator of model lightness and efficiency.

H. Combination of image and video

Image-only counting struggles with wide scenes and tiny objects, while video and multi-view methods extend coverage but retain assumptions about temporal data, camera placement, and calibration.

  • Video counting: Video methods leverage temporal consistency, including LSTM-based density estimation and constraints that conserve people across consecutive frames.These approaches extend counting beyond independent image predictions.
  • Wider-view crowd counting: Single-view counting cannot capture sufficient detail in large, wide scenes such as public parks or long subway platforms.Multi-view methods therefore predict 2D or 3D scene-level density maps.
  • Wider-view crowd counting: 3D feature fusion preserves 2D density-map properties while extracting crowd-density information along the height dimension.The cited comparison distinguishes 3D fusion from 2D fusion by its added z-dimension information.
  • Wider-view crowd counting: Existing multi-view approaches assume fixed cameras with known parameters, leaving moving cameras and unknown calibration as challenging future settings.This assumption defines an important boundary for wider-view counting systems.
  • Small or tiny objects: Tiny heads in congested scenes remain difficult, and related counting applications include dense buildings, ships, and small vehicles in remote-sensing images.The survey connects small-object counting to broader object-counting domains.
  • Conclusion: The survey benchmarks representative models, analyzes leading performers and performance attributes, and identifies open issues for future crowd counting research.It also frames these insights as a bridge to object counting in other domains.
Loading 2003.12783v1…