PaperlayerView exact revision on arXiv

Source-linked AI summary

JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method

Vishwanath A. Sindagi, Rajeev Yasarla, Vishal M. Patel

arXiv:2004.03597v2cs.CV

TL;DR

Single-image crowd counting needs large, diverse datasets and methods that handle challenging environmental conditions. The paper introduces JHU-CROWD++ and a confidence-guided progressive residual network, reporting comparable performance with state-of-the-art methods on recent datasets.

  • Problem

    Crowd-counting CNNs require substantial training data, while existing datasets have limited environmental diversity and annotation richness.

  • Method

    The paper constructs JHU-CROWD++ and uses progressive coarse-to-fine residual learning with uncertainty-based confidence weighting and class conditioning.

  • Results

    The proposed method achieves comparable performance with respect to state-of-the-art methods on recent datasets.

  • Takeaways & Limitations

    JHU-CROWD++ broadens crowd-counting evaluation with weather degradations, distractors, and richer image-level and head-level annotations.

Abstract

from arXiv · show

Due to its variety of applications in the real-world, the task of single image-based crowd counting has received a lot of interest in the recent years. Recently, several approaches have been proposed to address various problems encountered in crowd counting. These approaches are essentially based on convolutional neural networks that require large amounts of data to train the network parameters. Considering this, we introduce a new large scale unconstrained crowd counting dataset (JHU-CROWD++) that contains "4,372" images with "1.51 million" annotations. In comparison to existing datasets, the proposed dataset is collected under a variety of diverse scenarios and environmental conditions. Specifically, the dataset includes several images with weather-based degradations and illumination variations, making it a very challenging dataset. Additionally, the dataset consists of a rich set of annotations at both image-level and head-level. Several recent methods are evaluated and compared on this dataset. The dataset can be downloaded from http://www.crowd-counting.com . Furthermore, we propose a novel crowd counting network that progressively generates crowd density maps via residual error estimation. The proposed method uses VGG16 as the backbone network and employs density map generated by the final layer as a coarse prediction to refine and generate finer density maps in a progressive fashion using residual learning. Additionally, the residual learning is guided by an uncertainty-based confidence weighting mechanism that permits the flow of only high-confidence residuals in the refinement path. The proposed Confidence Guided Deep Residual Counting Network (CG-DRCN) is evaluated on recent complex datasets, and it achieves significant improvements in errors.

1 INTRODUCTION

The paper introduces JHU-CROWD++, a large-scale unconstrained dataset designed to address limited diversity, adverse conditions, and sparse annotations in crowd-counting benchmarks. It also proposes a residual-learning network with uncertainty-based confidence weighting and extends the method with class conditioning and backbone generalization.

  • Dataset motivation: Existing crowd-counting datasets have limitations in training-sample size, environmental diversity, dataset bias, and annotation richness.These gaps motivate collecting data under more varied real-world conditions and providing richer supervision.
  • Dataset contribution: JHU-CROWD++ contains 4,372 images and 1,515,005 head annotations collected under varied conditions, including weather degradations and distractor images.It provides image-level and head-level annotations covering points, approximate sizes, blur, occlusion, and weather labels.
  • Method contribution: The proposed network progressively generates improved coarse-to-fine density maps through residual learning based on a VGG16 backbone.Residual learning is used to refine density estimates rather than producing only a single prediction.
  • Method improvements: Uncertainty-based confidence weighting propagates only highly confident residuals, while class conditioning incorporates image-level weather information for adverse-weather counting.The extended method also evaluates whether uncertainty-based residual learning generalizes to a Res101 backbone.
  • Dataset improvements: The dataset extends JHU-CROWD with 122 additional images, 31% more annotations, and improved approximate width-and-height scale annotations.The annotation count increases from 1.15 million to 1.51 million.

2 RELATED WORK

Crowd-counting research has progressed from traditional representations and regression toward CNN-based methods addressing scale, density-map quality, attention, semantics, context, and multi-layer features. Related work therefore spans both architectural changes and incorporation of auxiliary information.

  • Traditional methods: Traditional single-image crowd-counting methods use hand-crafted representations with detection-, regression-, or density-estimation-based approaches.These categories organize earlier methods before the recent CNN-focused literature.
  • CNN-based counting: CNN-based methods have substantially improved crowd-counting performance and commonly address variation in head sizes across images.Examples include layered boosting, scale-aware architectures, and scale aggregation modules.
  • Alternative architectural strategies: Later methods target blurred density maps, dataset-conditioned capacity, cascaded density estimation, and deep negative-correlation learning.These approaches broaden the focus beyond handling head-size variation alone.
  • Auxiliary information: Other approaches incorporate attention, semantic priors, segmentation, inverse attention, hierarchical attention, perspective, and context information.These methods augment network representations with related visual or structural information.

3 JHU-CROWD++: LARGE-SCALE CROWD

JHU-CROWD++ addresses limited scale, environmental diversity, dataset bias, and annotation richness in existing crowd-counting datasets. It contributes a diverse unconstrained dataset with detailed labels and defined train/validation/test distributions.

  • Motivation: Existing datasets are limited by training-sample size, environmental diversity, dataset bias, and annotation breadth.These shortcomings affect sample diversity, robustness to adverse conditions, false crowd predictions, and access to head attributes such as size, blur, and occlusion.
  • Dataset: The dataset includes distractor images to address crowd-focused dataset bias.The 106 distractor images resemble crowd scenes but contain very few people, helping reduce the risk of predicting crowds in non-crowd scenes.
  • Annotations: Head-level annotations record locations, occlusion, blur, and approximate size, while image-level labels provide additional condition information.Occlusion has three levels, blur has two labels, and approximate size is represented by head width and height.
  • Collection and evaluation: Images are collected using diverse search keywords and split into 2,722 training, 500 validation, and 1,600 test images.The paper also reports distributions across density and weather categories for the train, validation, and test sets.

4 PROPOSED METHOD

CG-DRCN estimates crowd density progressively: a coarse VGG16-based map is refined with residuals from shallower features. Uncertainty-guided confidence weighting gates residual propagation, while class conditioning incorporates image-level weather labels.

  • Base network: CG-DRCN uses VGG16 layers C1–C5, followed by convolutional processing and max pooling, to produce a coarse density map with global context.The deepest prediction ˆY6 has low resolution and size W/32 × H/32.
  • Residual learning: Residual learning refines coarse density maps by adding learned residuals from shallower feature layers to up-sampled predictions.U-REB5 produces ˆR5, which is added to up(ˆY6) to generate ˆY5; subsequent blocks similarly produce ˆY4 and ˆY3.
  • Uncertainty-guided residual learning: U-REB predicts pixel-wise confidence for residuals and gates them before propagation, allowing only highly confident residuals to affect later outputs.Confidence estimation uses residual maps together with dimensionally reduced main-branch features.
  • Uncertainty-guided residual learning: The training objective combines density-map regression with confidence guidance so confidence scores and prediction accuracy are optimized jointly.Ld minimizes prediction error, while Lc encourages confidence scores toward 1; their interaction prevents confidence from being optimized independently of accurate density prediction.
  • Progressive refinement: The network’s density maps progressively improve in fine detail and count value from ˆY6 through ˆY3, and confidence gating improves feature learning.The architecture repeats refinement through U-REB5, U-REB4, and U-REB3.
  • Class-conditioned refinement: Class conditioning augments U-REB with image-level weather labels, conditioning uncertainty-guided residual estimation on weather information.The class-conditioning features are concatenated with residual and reduced main-branch features before confidence prediction.

5 ABLATION STUDY

Ablations show that residual learning improves crowd-counting performance, with further gains from confidence-guided residual weighting, class conditioning, multiscale estimation, and more training data.

  • Network components: Residual learning considerably improves performance over the base VGG16 network, and confidence-based residual weighting yields further improvements.The ablation evaluates a VGG16 base network, residual learning, and confidence-guided residual estimation.
  • Class-conditioning: Class-conditioning improves density-map quality and prediction error on weather-degraded images.The improvement is visualized in the weather-conditions ablation, with highlighted regions showing better estimates.
  • Backbone generalization: Residual learning, uncertainty-guided estimation, and class conditioning also improve performance with a Res101 backbone.The authors report that these observations mirror the VGG16 ablation and indicate generalization across architectures.
  • Multiscale estimation: Adding residual-estimation blocks at more network scales reduces errors for both VGG16 and Res101 architectures.Blocks are sequentially added at multiple layers to evaluate multiscale residual estimation.
  • Training data size: MAE decreases as the percentage of JHU-CROWD++ training samples increases from 5% to 100%.The experiment trains CG-DRCN-CC-Res101 on five dataset-size settings and evaluates on the validation set.

6 BENCHMARKING ON JHU-CROWD++ DATASET

On JHU-CROWD++, CG-DRCN-CC with Res101 achieves the strongest overall test performance and performs especially well on high-density and weather categories, while VGG16 remains comparable across categories.

  • Overall results: CG-DRCN-CC with Res101 achieves the lowest overall MAE/MSE among the compared methods on the JHU-CROWD++ test set.It also achieves the best errors for the high-density and weather categories.
  • Category results: The Res101 version obtains the best errors in the high-density and weather categories while remaining comparable in the other categories.The benchmark reports category-specific results across JHU-CROWD++.
  • Category results: CG-DRCN-CC with VGG16 achieves comparable performance across all categories relative to the other methods.The benchmark compares the proposed variants with several recent crowd-counting approaches.
  • Backbone comparison: Res101-based methods tend to perform better than VGG16-based approaches in overall error.The benchmark also notes that BCC and LSC-CNN achieve the lowest errors in low-density categories.

7 EVALUATION ON OTHER DATASETS

The proposed method is evaluated on ShanghaiTech and UCF-QNRF, outperforming existing methods on ShanghaiTech Part A, matching them comparably on Part B, and achieving comparable performance on UCF-QNRF.

  • ShanghaiTech: ShanghaiTech contains 1,198 annotated images and 330,165 people across Parts A and B.Part A has 482 images and Part B has 716, with separate training and test splits.
  • ShanghaiTech: The proposed method outperforms all existing methods on ShanghaiTech Part A while achieving comparable performance on Part B.The comparison includes multiple recent crowd-counting methods.
  • UCF-QNRF: On UCF-QNRF, the proposed method achieves comparable performance relative to recent state-of-the-art methods.UCF-QNRF contains 1,535 high-resolution images and 1.25 million head annotations.

8 CONCLUSIONS

The paper introduces JHU-CROWD++ and a progressive residual crowd-counting network with uncertainty-guided confidence weighting and class conditioning, then benchmarks the method on multiple datasets.

  • Conclusions: JHU-CROWD++ contains 4,372 images and 1.51 million annotations collected under varied conditions, including weather degradations and distractor images.The dataset includes head locations, blur and occlusion levels, approximate bounding boxes, and image-level labels.
  • Conclusions: The proposed network progressively estimates coarse-to-fine density maps using residual learning and propagates only high-confidence residuals through uncertainty-based weighting.Class conditioning uses image-level labels to improve performance in adverse weather conditions.
  • Conclusions: The method is evaluated on recent datasets and achieves comparable performance relative to state-of-the-art methods.The conclusion summarizes the method's benchmarking across the proposed and external datasets.
Loading 2004.03597v2…