PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating High-Quality Crowd Density Maps using Contextual Pyramid CNNs

Vishwanath A. Sindagi, Vishal M. Patel

arXiv:1708.00953v1cs.CV

TL;DR

Crowd counting and density estimation must handle complex scenes while producing maps that are more than merely count-accurate. CP-CNN combines global and local contextual classifiers with a multi-column density estimator and Fusion-CNN, using adversarial and Euclidean losses; experiments report significant improvements over state-of-the-art methods, with computational cost as a limitation.

  • Problem

    Crowd-estimation methods face complex visual conditions and often produce low-quality maps or biased counts across density levels.

  • Method

    CP-CNN combines global and patch-wise local context estimators with a multi-column Density Map Estimator and Fusion-CNN trained using weighted adversarial and pixel-wise Euclidean losses.

  • Results

    Experiments on three challenging datasets and ablations report significant improvements over state-of-the-art methods in count estimation and density-map quality.

  • Takeaways & Limitations

    The paper supports explicitly modeling context and optimizing map quality alongside count error within crowd-density estimation.

  • Takeaways & Limitations

    Additional context processing makes inference computationally expensive compared with earlier methods.

Abstract

from arXiv · show

We present a novel method called Contextual Pyramid CNN (CP-CNN) for generating high-quality crowd density and count estimation by explicitly incorporating global and local contextual information of crowd images. The proposed CP-CNN consists of four modules: Global Context Estimator (GCE), Local Context Estimator (LCE), Density Map Estimator (DME) and a Fusion-CNN (F-CNN). GCE is a VGG-16 based CNN that encodes global context and it is trained to classify input images into different density classes, whereas LCE is another CNN that encodes local context information and it is trained to perform patch-wise classification of input images into different density classes. DME is a multi-column architecture-based CNN that aims to generate high-dimensional feature maps from the input image which are fused with the contextual information estimated by GCE and LCE using F-CNN. To generate high resolution and high-quality density maps, F-CNN uses a set of convolutional and fractionally-strided convolutional layers and it is trained along with the DME in an end-to-end fashion using a combination of adversarial loss and pixel-level Euclidean loss. Extensive experiments on highly challenging datasets show that the proposed method achieves significant improvements over the state-of-the-art methods.

1. Introduction

Crowd counting and density estimation remain difficult under occlusion, clutter, non-uniformity, and appearance, scale, illumination, and perspective variation. CP-CNN addresses these challenges by combining global and local context with density-map estimation to improve count accuracy and map quality.

  • Crowd analysis supports surveillance, traffic monitoring, public safety, urban planning, scene understanding, and related counting applications.
  • Occlusions, clutter, non-uniform distributions, illumination changes, and appearance, scale, and perspective variation keep crowd-estimation accuracy from being optimal.
  • Multi-scale CNN methods can over-estimate low-density counts and under-estimate high-density counts, motivating contextual information during learning.
  • Existing methods often produce low-resolution density maps and emphasize count quality over density-map quality, prompting CP-CNN to target both.
  • CP-CNN combines VGG-16-based global context estimation, patch-wise local context estimation, multi-column feature extraction, and Fusion-CNN density-map reconstruction.
  • The method uses fractionally-strided convolutions and a weighted Euclidean-plus-adversarial loss to generate high-quality maps, with experiments and ablations across three challenging datasets.

2. Related work

Prior crowd-counting research includes detection, segmentation, regression, density-estimation, and CNN-based approaches, but existing methods leave contextual modeling and density-map quality insufficiently addressed. CP-CNN builds on this literature by explicitly leveraging context at multiple levels and adding adversarial training.

  • Early detection- and segmentation-based crowd methods were adversely affected by occlusions and high background clutter.
  • CP-CNN introduces global and local context estimators alongside a Density Map Estimator and Fusion-CNN to produce high-resolution, high-quality density maps.
  • CNN-based work progressed from layered and patch-based methods toward end-to-end models and multi-column architectures for crowd counting and density estimation.
  • Unlike cascaded high-level-prior methods, CP-CNN targets over- and under-estimation by systematically using crowd-density context at multiple levels.
  • Existing approaches generally lack explicit contextual information, prioritize count error over map quality, and use Euclidean loss associated with blurred density maps.

3. Proposed method (CP-CNN)

CP-CNN combines global and local context estimators with a multi-column density-map estimator and Fusion-CNN to produce high-resolution, high-quality crowd density maps. Its training uses contextual classification, feature fusion, and a weighted Euclidean–adversarial objective.

  • Architecture: CP-CNN comprises Global Context Estimator, Local Context Estimator, Density Map Estimator, and Fusion-CNN modules.GCE and LCE encode global and local context, DME produces high-dimensional features, and F-CNN fuses them into density maps.
  • Global Context Estimator (GCE): GCE uses a VGG-16-based network to classify whole images into density categories for global context encoding.The network is fine-tuned for five density categories, with earlier convolutional weights retained.
  • Local Context Estimator (LCE): LCE classifies local image patches into density categories to encode spatially local context for density-map estimation.The five categories are ex-lo, lo, med, hi, and ex-hi.
  • Density Map Estimator (DME): DME uses a multi-column CNN with deeper columns and varied filter configurations to handle substantial variation in head sizes.Its high-dimensional feature maps are concatenated with contextual information before Fusion-CNN processing.
  • Fusion-CNN (F-CNN): Fusion-CNN combines DME features with GCE and LCE context, using fractionally-strided convolutions to restore spatial detail and output resolution.Each fractionally-strided convolution doubles input resolution, while F-CNN restores details lost through DME max-pooling.
  • Training objective: DME and F-CNN are trained end-to-end with a weighted combination of pixel-wise Euclidean loss and adversarial loss.The adversarial term is introduced alongside Euclidean loss to improve density-map quality.

4. Training and evaluation details

Training first establishes global and local context estimators, then trains DME and F-CNN end-to-end using contextual feature maps. At inference, image-level and patch-level context maps are combined with DME features, although the added processing makes inference computationally expensive.

  • Training data: Training creates 300 augmented quarter-size patches per image for DME and separately crops 64 × 64 patches for LCE.The DME set uses random cropping, horizontal flipping, and noise addition; LCE uses patches from 100 random locations per image.
  • Context-estimator training: GCE is trained on image-level density categories, while LCE is trained on 64 × 64 patches labeled by their people counts.GCE inputs are resized to 224 × 224 and trained with standard cross-entropy loss.
  • End-to-end training: DME and F-CNN are trained end-to-end after context estimation, minimizing the weighted Euclidean–adversarial loss between estimated and ground-truth density maps.Global and local context maps are concatenated with DME feature maps before F-CNN produces the output.
  • Context-map construction: Global context maps are formed by classifying the input image and filling five feature maps with the resulting density-class scores.The global context representation has dimensions 5 × W_i/4 × H_i/4.
  • Inference: Inference computes global and local context maps, concatenates them with DME features, and feeds the combined representation into F-CNN.GCE processes non-overlapping image blocks, while LCE runs across sliding windows.
  • Inference: Additional context processing makes CP-CNN inference computationally expensive compared with earlier methods such as [29].This is identified as a practical cost of the proposed inference procedure.

5. Experimental results

Experiments evaluate CP-CNN through ablations and comparisons on ShanghaiTech, WorldExpo’10, and UCF CC 50. Contextual modules, fractionally-strided convolutions, and adversarial loss improve density-map quality and count estimation, with strong benchmark results.

  • Ablation study: The ablation study evaluates DME, contextual modules, Fusion-CNN, and adversarial loss against Zhang et al. on ShanghaiTech Part A.Each module is added sequentially, and the complete network uses local and global context with LA + LE.
  • Ablation study: Global context greatly reduces count error, while Fusion-CNN improves density-map quality through fractionally-strided convolutional layers.The DME architecture changes improve count error over Zhang et al., but do not significantly improve density-map quality by themselves.
  • Ablation study: Adding local context and adversarial loss progressively reduces count error and improves PSNR and SSIM density-map quality.Visual comparisons likewise report better estimation quality after adding contextual information and optimizing the weighted adversarial and Euclidean losses.
  • Benchmark comparisons: The proposed method achieves superior results against recent methods on the ShanghaiTech dataset.The comparison is reported in Table 2 for both Part A and Part B.
  • Benchmark comparisons: On WorldExpo’10, CP-CNN outperforms existing approaches on average while achieving comparable performance in individual scene estimations.The dataset contains 600 test frames from five scenes, and average estimation errors are reported in Table 3.
  • Benchmark comparisons: On UCF CC 50, the network achieves the lowest MAE and MSE count errors across five-fold cross-validation.The experiment uses 50 images with widely varying densities and compares CP-CNN with seven recent approaches.

6. Conclusion

The conclusion presents CP-CNN as a contextual pyramid of CNNs for crowd counting and high-quality density-map generation. It combines learned global and local context with DME and Fusion-CNN training, and reports significant improvements on challenging datasets.

  • Conclusion: CP-CNN incorporates global and local contextual information to generate high-quality crowd density maps and lower count estimation errors.The method learns context from image- and patch-level density categories.
  • Conclusion: Global and local context are fused with multi-column DME features through a Fusion-CNN using fractionally-strided convolutional layers.The DME and Fusion-CNN are trained end-to-end with a weighted combination of adversarial and pixel-wise Euclidean loss.
  • Conclusion: Experiments on challenging datasets and comparisons with recent state-of-the-art methods demonstrate significant improvements from the proposed method.

Appendix

The appendix provides representative sample results for ShanghaiTech, UCF CC 50, and WorldExpo ’10. Each visualization contrasts inputs, ground-truth density maps, and estimated density maps across density levels.

  • Additional results: Sample images from all three evaluated datasets were selected to represent various density levels.The datasets are ShanghaiTech, UCF CC 50, and WorldExpo ’10.
  • Additional results: For ShanghaiTech Part A, each result presents the input image, ground-truth density map, and estimated density map in left-to-right columns.
  • Additional results: For ShanghaiTech Part B, each result presents the input image, ground-truth density map, and estimated density map in left-to-right columns.
  • Additional results: For UCF CC 50, each result presents the input image, ground-truth density map, and estimated density map in left-to-right columns.
  • Additional results: For WorldExpo ’10, each result presents the input image, ground-truth density map, and estimated density map in left-to-right columns.
Loading 1708.00953v1…