PaperlayerView exact revision on arXiv

Source-linked AI summary

PCC Net: Perspective Crowd Counting via Spatial Convolutional Network

Junyu Gao, Qi Wang, Xuelong Li

arXiv:1905.10085v1cs.CVcs.AI

TL;DR

Single-image crowd counting is challenged by appearance similarity, perspective changes, severe congestion, and methods that emphasize local features. PCC Net combines density estimation, random high-level density classification, foreground/background segmentation, and four-directional perspective encoding, with reported competitive results across five datasets and especially improved performance in extremely congested scenes.

  • Problem

    Existing crowd-counting methods struggle with high appearance similarity, complex perspective changes, severe congestion, and background regions that resemble dense crowds.

  • Method

    PCC Net uses DME for local density estimation, R-HDC for global random-patch density classification, FBS for foreground/background segmentation, and DULR for perspective encoding.

  • Results

    The authors report competitive results on five datasets and significant counting improvements for extremely congested crowd scenes.

  • Takeaways & Limitations

    PCC Net combines hierarchical features and perspective changes to improve density-map quality and counting in congested scenes.

Abstract

from arXiv · show

Crowd counting from a single image is a challenging task due to high appearance similarity, perspective changes and severe congestion. Many methods only focus on the local appearance features and they cannot handle the aforementioned challenges. In order to tackle them, we propose a Perspective Crowd Counting Network (PCC Net), which consists of three parts: 1) Density Map Estimation (DME) focuses on learning very local features for density map estimation; 2) Random High-level Density Classification (R-HDC) extracts global features to predict the coarse density labels of random patches in images; 3) Fore-/Background Segmentation (FBS) encodes mid-level features to segments the foreground and background. Besides, the DULR module is embedded in PCC Net to encode the perspective changes on four directions (Down, Up, Left and Right). The proposed PCC Net is verified on five mainstream datasets, which achieves the state-of-the-art performance on the one and attains the competitive results on the other four datasets. The source code is available at https://github.com/gjy3035/PCC-Net.

I. INTRODUCTION

PCC Net addresses crowd-counting errors from background confusion, complex perspective changes, and limited global context through multi-task learning and perspective encoding.

  • Problem: Crowd counting generates a density map whose pixel-value sum predicts the number of people in a scene.The task is motivated by applications including video surveillance, public safety, urban planning, and behavior understanding.
  • Problem: High clutter, appearance similarity, and perspective changes make existing methods prone to background misestimation and poor density maps.Tiny heads in highly congested areas are also difficult for detection-based methods to identify accurately.
  • Method: The DULR module encodes perspective changes along Down, Up, Left, and Right directions, particularly for congested scenes.The authors report that this perspective module improves density estimation in congested crowd scenes.
  • Results: PCC Net outputs more perspective-consistent density trends than CSRNet and avoids some background-tree errors in challenging examples.The comparison figure orders original images, ground truth, PCC Net results, and CSRNet results by columns.
  • Results: The paper reports that PCC Net significantly improves performance over CSRNet in the illustrated challenging cases.The cited comparison describes PCC Net as closer to ground truth for uniform perspective and better aligned with perspective density trends.
  • Method: PCC Net combines Density Map Estimation, Random High-level Density Classification, and Fore-/Background Segmentation for local, global, and region-level information.R-HDC classifies randomly sampled image regions with coarse density labels during training, producing more diversified samples than a fixed training set.

II. RELATED WORK

Related work spans patch-based and whole-image crowd counting, image segmentation, and spatial CNNs; PCC Net combines these directions through FCN-based density and segmentation tasks.

  • Patch-based crowd counting: Patch-based methods crop images at different sizes to capture local features and generate training data, but can perform poorly on unseen target scenes.Some methods classify image density, regress counts, or fine-tune models on target scenes.
  • Whole image-based crowd counting: Whole-image methods address the limited global context of patch-based approaches using architectures such as multi-column CNNs, CNN-LSTM models, and semantic-spatial representations.These methods process complete images or incorporate broader contextual information for counting.
  • Image segmentation and spatial CNN: Image segmentation research introduced FCNs and related architectures that combine semantic, contour, contextual, and spatial information.Spatial CNN research includes dilated convolution, affinity-based propagation, and slice-by-slice convolutional operations.
  • PCC Net: The PCC Net architecture contains DME, R-HDC, and FBS streams, as shown in its overview figure.The figure identifies the three named components but does not provide additional performance results.
  • PCC Net: PCC Net uses FCNs for pixel-wise density regression and foreground/background classification, while R-HDC performs random-patch high-level classification.DULR is added to the FCN streams to encode perspective changes.
  • Density Map Estimation: DME is designed as a simple FCN that predicts crowd density maps from feature-map flow without a classification layer.Its training loss is standard Mean Squared Error.

B. Random High-level Density Classification

R-HDC addresses the limits of local-only density estimation by learning global features from randomly generated image regions with coarse density labels. The broader framework adds segmentation features to preserve larger-scale head and contextual information.

  • Motivation: FCN-based density estimation focuses on local features but ignores global context, limiting its ability to reduce mistaken density estimates.
  • Random High-level Density Classification: R-HDC is an end-to-end strategy that randomly generates ROI patches during training instead of using a fixed precomputed patch set.Random regions are extracted from whole images and assigned coarse density labels online.
  • Random High-level Density Classification: R-HDC formulates high-level density prediction as a 10-class classification problem based on people count divided by image area.
  • Limitations: The DME+R-HDC combination extracts local and global features, but can neglect head shape in sparse regions and context in congested regions.
  • Fore-/Background Segmentation: FBS addresses these omissions by using larger head-region masks that contain head shape, facial structure, and crowd-distribution context.Coarse masks are generated by dilating key-point binary images with a ball-shaped structuring element of radius and height 50 pixels.
  • Fore-/Background Segmentation: DME and FBS share a base feature extractor, with the segmentation stream appended to the DME stream for density-map estimation.Their losses use different scales, making direct joint training difficult; FBS uses standard 2-D cross-entropy loss.

D. Perspective Encoder: DULR Module

The DULR module uses spatial convolutional processing in four directions to aggregate spatial information across feature maps. Its directional aggregation reflects perspective-dependent differences across image rows and columns.

  • Architecture: DULR uses four spatial convolutional layers—Down, Up, Left-to-right, and Right-to-left—to encode perspective changes.
  • Directional processing: The feature map is divided into directional parts, processed recurrently through convolution and ReLU, and concatenated into an output with the original spatial size.For the Down Layer, successive parts are combined iteratively before producing the corresponding output parts.
  • Directional processing: The Down Layer is defined as a convolution plus ReLU, with the H feature-map parts sharing the same layer.
  • Perspective encoding: Each row’s result aggregates information from rows above it, while differing computation orders produce row-specific aggregation effects.
  • Perspective encoding: DULR preserves the input feature-map size while introducing global spatial information across the image, particularly for extremely congested regions.

IV. EXPERIMENTS

The experiments report evaluation metrics and experimental details, include ablation studies on ShanghaiTech Part A, and evaluate PCC Net on four mainstream datasets.

  • Experimental design: The evaluation includes metric definitions, implementation details, ShanghaiTech Part A ablations, and results on four mainstream datasets.

A. Evaluation

Evaluation uses counting errors and density-map image-quality measures. The implementation fixes training settings and uses PyTorch on an NVIDIA GTX 1080Ti GPU.

  • Metrics: MAE and MSE are the mainstream metrics used to evaluate crowd-counting performance.MAE and MSE are defined using test-set counts and estimated count values.
  • Metrics: PSNR and SSIM evaluate the quality of predicted density maps.
  • Implementation details: PCC Net uses fixed learning rates, batch size 12, 20 R-HDC ROIs, kernel width ω=1, and images resized to 576 × 768.
  • Implementation details: Training and evaluation use PyTorch on an NVIDIA GTX 1080Ti GPU.

2) Data augmentation:

The ablation study evaluates step-wise PCC Net variants on ShanghaiTech Part A, including DME, R-HDC, FBS, and DULR additions. The full model achieves the strongest reported errors, while R-HDC is compared with fixed-patch HDC.

  • The Results of Step-wise Experiments: ShanghaiTech Part A ablations compare DME, DME+R-HDC, DME+R-HDC+FBS, and the full DME+R-HDC+FBS+DULR model.The study uses 300 training images and 182 testing images with different sizes.
  • The Results of Step-wise Experiments: Each added step significantly improves performance, with PCC Net achieving the best MAE (73.5) and MSE (124.0).The full model adds DULR to the DME and FBS architectures.
  • The Results of Step-wise Experiments: Visualization shows progressively improved density maps, with FBS reducing background errors and DULR producing closely uniform perspective-aware density maps.The examples include slight-perspective and congested crowd scenes.
  • Data augmentation: R-HDC records MAE 95.6 and MSE 147.6, compared with HDC's MAE 88.9 and MSE 137.2 in the reported comparison.The passage attributes R-HDC's learning advantage to iterative coverage and random-size feature-map crops.

3) The Effect of DULR Module:

The DULR ablation replaces its directional operations with four standard convolutions and evaluates PCC Net across ShanghaiTech and WorldExpo’10. Results indicate strong performance on ShanghaiTech, while DULR has limited effect in low-density WorldExpo’10 scenes where perspective maps are provided.

  • The Effect of DULR Module: Replacing DULR with four standard convolutions removes directional spatial encoding for perspective changes.The 4conv baseline uses the same kernel size as DULR but removes slice-by-slice operations in four directions.
  • The Effect of DULR Module: DULR reduces MAE from 80.1 to 73.5 and MSE from 124.4 to 124.0, whereas 4conv changes MAE to 79.4 and increases MSE to 130.6.These comparisons are reported for DME+R-HDC+FBS with either 4conv or DULR.
  • Results on ShanghaiTech: On ShanghaiTech Part B, PCC Net achieves MAE 11.0 and MSE 19.0, improving over ACSCP by 6.2 and 8.4 points, respectively.On Part A, it achieves MAE 73.5 and the second-best MSE 124.0 among the listed comparisons.
  • Results on WorldExpo’10: On WorldExpo’10, PCC Net achieves the best MAE of 1.9 on Scene1 and ranks third among no-pre-trained methods on average.The passage states that DULR's effect is limited in low-density scenes because the dataset supplies perspective maps.

F. Results on UCF CC 50

On UCF CC 50, PCC Net is evaluated with standard 5-fold crossvalidation and substantially outperforms competing methods, especially CSRNet, on extremely congested scenes.

  • Dataset and protocol: UCF CC 50 contains 50 images with 63,075 labeled individuals and counts ranging from 94 to 4,543.The dataset covers diverse scenes with varying perspective distortion.
  • Dataset and protocol: Standard 5-fold crossvalidation is used because the dataset contains only 50 images.
  • Quantitative results: 26.1 points in MAE and 82.0 points in MSE improve over CSRNet on UCF CC 50.The authors report that these gains demonstrate effectiveness on extremely congested crowd scenes.
  • Additional benchmark context: PCC Net achieves the best reported result among three no-pretrained methods on UCF-QNRF, with MAE 148.7 and MSE 247.3.It outperforms Switching-CNN but remains above CL, which reports MAE 132 and MSE 191.
  • Efficiency comparison: On ShanghaiTech Part A, PCC Net is best except for MSE and uses 2.0MB, 0.55M parameters, and 89ms runtime.The runtime is compared with 153ms for Switching-CNN and 5113ms for CP-CNN on one NVIDIA GTX 1080Ti GPU.

B. Analysis of the Loss Weights

The loss-weight study finds the best MAE and MSE when λ and β are both 0.0001, with β affecting density estimation more strongly than λ.

  • Loss-weight selection: 0.0001 for both λ and β yields the best MAE and MSE in the tested loss-weight settings.The tested values are 1, 10^-2, 10^-4, and 10^-6.
  • Loss-weight effects: β affects density estimation more dramatically than λ.The authors relate this to greater convolutional-layer sharing between FBS and DME than between R-HDC and DME.
  • Loss-weight effects: DME receives a larger weight because regression is considered more difficult than R-HDC and FBS classification tasks.
  • Perspective-map analysis: DME+DULR predicts perspective maps better than DME+4conv despite using the same number of parameters.
  • Perspective-map analysis: DME+DULR produces higher-quality perspective maps and captures gradual perspective changes, whereas DME+4conv is sensitive to local features.

D. Selection of the ROI’s Number

The ROI study examines random patch coverage and training convergence, selecting 20 ROIs because more ROIs accelerate R-HDC convergence without improving overall classification or DME training sufficiently.

  • ROI generation: R-HDC randomly generates patches to cover the image rather than assign real object categories.Iterative training theoretically produces 1,600 different patches per image over 800 epochs.
  • ROI coverage: At least 16 non-overlapping patches would be needed to cover an image when each patch is 1/16 of the input area.Random placement cannot guarantee complete image coverage in theory.
  • Convergence analysis: More ROIs produce faster R-HDC convergence, but similar cross-entropy curves are observed for 5, 20, and 300 ROIs.The broader tested groups are also described as close to one another.
  • Final selection: 20 ROIs are selected because faster R-HDC convergence does not significantly accelerate DME training.

E. Comparative Visualization Results with SOTA

Comparative visualizations show PCC Net reduces background misestimation and better preserves perspective consistency than CP-CNN and CSRNet, while foreground segmentation depends on suitable mask settings.

  • Visualization setup: Fig. 9 arranges input images, ground-truth density maps, and results from PCC Net, CP-CNN, and CSRNet in separate rows.The red boxes highlight selected differences between the methods.
  • Background regions: CP-CNN and CSRNet often mistake trees or plants for crowds, whereas PCC Net performs better in these background regions.The authors attribute this to discriminative foreground-background features.
  • Perspective consistency: PCC Net shows better perspective consistency across the whole image than CP-CNN and CSRNet.
  • Foreground-background segmentation: FBS uses morphology-generated head masks because the original datasets lack head segmentation masks.
  • Mask-parameter sensitivity: 10 to 60 pixels produces similar counting results, while values below 10 or above 100 pixels approach the NoSeg performance.The authors explain that extreme settings fail to cover valid foreground and background regions effectively.
  • Method interpretation: PCC Net combines hierarchical features and perspective changes through DME, R-HDC, FBS, and DULR.DME learns local density features, R-HDC predicts coarse density labels for random patches, and FBS segments head and background regions.
Loading 1905.10085v1…