PaperlayerView exact revision on arXiv

Source-linked AI summary

NWPU-Crowd: A Large-Scale Benchmark for Crowd Counting and Localization

Qi Wang, Junyu Gao, Wei Lin, Xuelong Li

arXiv:2001.03360v4cs.CV

TL;DR

Existing crowd counting datasets are too small and lack complete localization labels and impartial evaluation, limiting support for supervised CNN-based methods. The paper constructs NWPU-Crowd, adds a benchmark, and evaluates representative methods, finding errors on negative samples and interactions across scene attributes. The dataset also exposes challenges in robustness, appearance variation, extreme congestion, and jointly localizing tiny and large heads.

  • Problem

    Existing crowd datasets are too small for CNN-based methods, often lack box-level localization labels, and lack an impartial evaluation benchmark.

  • Method

    The paper constructs NWPU-Crowd, develops an online benchmark, deduplicates data using VGG-16 feature similarity, and adds box annotations through regression followed by manual refinement.

  • Results

    The dataset contains 5,109 images and 2,133,375 annotated instances, while experiments reveal estimation errors on negative samples and significant interactions between density and luminance.

  • Takeaways & Limitations

    NWPU-Crowd provides a large real-world basis for counting and localization research while highlighting robustness, appearance representation, congestion, and scale-related challenges.

  • Takeaways & Limitations

    Existing models perform poorly in high-density regions because of head occlusions, small objects, and limited structured information.

Abstract

from arXiv · show

In the last decade, crowd counting and localization attract much attention of researchers due to its wide-spread applications, including crowd monitoring, public safety, space design, etc. Many Convolutional Neural Networks (CNN) are designed for tackling this task. However, currently released datasets are so small-scale that they can not meet the needs of the supervised CNN-based algorithms. To remedy this problem, we construct a large-scale congested crowd counting and localization dataset, NWPU-Crowd, consisting of 5,109 images, in a total of 2,133,375 annotated heads with points and boxes. Compared with other real-world datasets, it contains various illumination scenes and has the largest density range (0~20,033). Besides, a benchmark website is developed for impartially evaluating the different methods, which allows researchers to submit the results of the test set. Based on the proposed dataset, we further describe the data characteristics, evaluate the performance of some mainstream state-of-the-art (SOTA) methods, and analyze the new problems that arise on the new data. What's more, the benchmark is deployed at \url{https://www.crowdbenchmark.com/}, and the dataset/code/models/results are available at \url{https://gjy3035.github.io/NWPU-Crowd-Sample-Code/}.

1 INTRODUCTION

NWPU-Crowd addresses limitations in existing crowd datasets by providing large-scale point- and box-level annotations, negative samples, and impartial evaluation. Experiments reveal robustness and cross-scene representation challenges that remain important for crowd counting and localization.

  • Existing crowd datasets are too small for CNN-based methods, which require many labeled examples and may overfit limited data.
  • Traditional datasets lack box-level labels for uniform localization evaluation and do not provide an impartial benchmark for test-set comparison.
  • NWPU-Crowd contains 5,109 images and 2,133,375 annotated instances, with negative samples and counts ranging from 0 to 20,033.
  • Experiments show that diverse data makes it difficult for counting networks to learn useful and distinguishable features.
  • Negative samples produce many estimation errors, while density and luminance attributes significantly influence one another.
  • The authors expect the dataset to promote practical crowd counting and localization applications and encourage work on these problems.

2 RELATED WORKS

Related datasets span surveillance, general, synthetic, and specialized settings, but differ in scale, diversity, annotations, and realism. NWPU-Crowd is presented within this landscape as a large real-world dataset covering varied crowd conditions.

  • Surveillance view: Surveillance-view datasets target specific indoor or small-area outdoor scenes, typically containing between 0 and 600 people.
  • Synthetic data: Synthetic GCC provides accurate point and mask labels across diverse environments, but synthetic-to-real domain gaps limit practical value.
  • Specialized datasets: Other datasets specialize in scenes, RGBD images, temporal sequences, or large views such as stadiums and plazas.
  • General-scene datasets: General-scene datasets collected online exhibit wide density variation, diverse scenarios, changing illumination, and uneven crowd distributions.
  • General-scene datasets: UCF CC 50 contains only 50 images, motivating larger datasets for robust deep learning models.
  • General-scene datasets: UCF-QNRF targets extreme congestion with 1,525 images and more than 1,251,642 label points, while Crowd Surveillance contains 13,945 images with regions of interest.

3 NWPU-CROWD DATASET

NWPU-Crowd is constructed as a large-scale, diverse crowd counting and localization dataset with point and box annotations, supported by cleaning, annotation, and evaluation protocols. Its scale, appearance variation, negative samples, and adaptive localization metrics target limitations in existing datasets and benchmarks.

  • Data collection and cleaning: Raw data are collected through self-shooting and Internet searches, then deduplicated with VGG-16 feature distances and filtered to 5,109 valid images.Images with D(ix, iy) < 5 are treated as similar, and one of each pair is removed; blurred images are also discarded.
  • Annotation: The annotation tool supports point and bounding-box labels, multiscale zooming, and division into up to 16 × 16 blocks for annotating heads at five scales.A regression procedure estimates box sizes for point-only heads using eight nearest box-labeled neighbors.
  • Data characteristics: 5,109 images and 2,133,375 annotated instances make NWPU-Crowd the largest dataset at both image and instance levels among compared crowd-counting datasets.The dataset includes normal-light, extreme-light, dark-light, and negative samples.
  • Data characteristics: 351 negative samples, average resolution 2191 × 3209, and counts from 0 to 20,033 broaden robustness, resolution, and appearance coverage.Negative samples include animal migrations, fake crowds, empty halls, and densely arranged non-person objects; the largest image is 4028 × 19044 pixels.
  • Data split and counting evaluation: The dataset is split into 3,109 training, 500 validation, and 1,500 test images, with counting evaluated by MAE, MSE, and NAE across scene and luminance classes.NAE excludes negative samples to avoid zero denominators.
  • Localization evaluation: Localization evaluation uses adaptive thresholds based on real head size and reports Recall separately across six box-area categories.The scheme includes stricter and looser thresholds and is intended to support deeper scale-specific analysis.

4 EXPERIMENTS ON COUNTING

Experiments evaluate mainstream counting methods on NWPU-Crowd and analyze validation/test behavior, scene difficulty, negative samples, and training-data scale. The results expose failures on hard samples while showing benefits from diverse data and negative-sample training.

  • Evaluation setup: Ten mainstream open-sourced methods are trained on NWPU-Crowd and evaluated through the benchmark.The experiments include quantitative results and validation-set visualizations.
  • Validation results: SCAR achieves the best validation counting performance, with MAE of 81.57 and MSE of 397.92.
  • Validation results: Negative samples and extremely congested or extreme-luminance scenes produce substantial estimation errors across current models.SCAR remains about 1,900 and 8,000 people from the labels on two extremely congested images.
  • Test results: On the test set, BL, SFCN†, and CANNet occupy the top three positions by overall MAE, while validation-set ordering changes significantly.Overall MAE is the primary ranking key.
  • Scene effects: Training separate density classes reduces MAE by 36.6%, 25.7%, 22.2%, and 12.7% across the four classes compared with joint training.The authors attribute this to diverse scenes and large appearance variations, especially head-scale variation.
  • Negative samples: Adding negative samples decreases category-wise MAE by 21.7%, 2.0%, 6.1%, and 17.4% in the reported examples.The authors relate this improvement to learning more discriminative features from diverse background objects.
  • Data volume: Using 100% rather than 10% of the training data reduces validation MAE from 158.35 to 105.79, a relative decrease of 33.2%.Validation errors decrease overall as training-data volume increases.

5 EXPERIMENTS ON LOCALIZATION

The experiments compare four crowd-localization methods on validation and test data using localization and counting metrics. Results show method-specific trade-offs, persistent failures on difficult scales and scenes, and no method handles all localization conditions well.

  • 5.1 Experimental Setup: Four crowd-localization methods are evaluated quantitatively and qualitatively using box-level labels.The evaluation covers localization and counting performance on the validation set, with test-set leaderboard results reported separately.
  • 5.2 Results Analysis on the Validation Set: σs localization results are worse than σl because σs imposes the stricter criterion.For each head, σs is less than σl.
  • 5.2 Results Analysis on the Validation Set: RAZ Loc produces the best localization result, while Faster RCNN has higher Precision but misses many objects.Detection-based methods have the poorest counting performance across the reported plans.
  • 5.2 Results Analysis on the Validation Set: In congested scenes, Faster RCNN misses 95.4% of heads, while VGG+GPR and RAZ Loc obtain good results but produce some background false positives.TinyFaces yields many false positives and produces more false positives than the other methods.
  • 5.2 Results Analysis on the Validation Set: No method handles crowd localization well across all conditions: detectors miss small objects, TinyFaces produces false positives, and other methods struggle with scale variation and background errors.These limitations span general detection, TinyFaces, and regression/classification approaches.
  • 5.3 Leaderboard: On the test set, RAZ Loc ranks first by Overall F1-measure under σl, but all methods perform poorly for tiny heads.Detection-based methods are better for extremely large heads, whereas regression/classification methods use small labels that cover limited semantic head regions.

6 CONCLUSION AND OUTLOOK

The paper concludes that NWPU-Crowd and its benchmark support broad evaluation of crowd counting and localization while exposing unresolved challenges. It identifies robustness, scene variation, extreme congestion, and joint localization of tiny and large heads as priorities for future work.

  • Conclusion: NWPU-Crowd contains high-resolution images, negative samples, and large appearance variation, alongside an online benchmark for fair model evaluation.The paper evaluates fourteen typical algorithms across counting, localization, density-map quality, and time complexity.
  • Conclusion: The experiments reveal new problems involving model robustness, interactions between different scene attributes, extreme congestion, and large-range scale variation.These issues are identified from quantitative and qualitative results on the proposed dataset.
  • Outlook: Negative samples are vital for measuring robustness because counters may incorrectly estimate unseen background regions.The paper links negative-sample performance with robustness in real-world counting.
  • Outlook: Training on all scene categories causes an obvious performance reduction compared with training separately on each category.The paper attributes this issue to large appearance variations and calls for stronger appearance representations.
  • Outlook: Existing methods struggle in high-density regions because of head occlusions, small objects, and lack of structured information.The paper also identifies accurate joint localization of tiny and large heads as an unresolved scale-variation problem.
  • Outlook: Future work will address these issues to improve crowd counting and localization performance in the real world.The paper specifically points toward handling the identified problems in subsequent research.
Loading 2001.03360v4…