Source-linked AI summary
Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework
Qingyu Song, Changan Wang, Zhengkai Jiang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Yang Wu
TL;DR
The paper addresses the difficulty of jointly counting and precisely localizing individuals when existing methods rely on density maps or pseudo boxes. It proposes a purely point-based framework with nAP evaluation and P2PNet, whose one-to-one matching supports direct point prediction. The framework and P2PNet achieve state-of-the-art counting performance and promising localization accuracy.
Problem
Existing crowd methods relying on density maps or pseudo bounding boxes do not directly provide precise individual locations and can be error-prone.
Method
The paper proposes a purely point-based framework, the nAP metric, and P2PNet with direct point proposals trained through one-to-one target matching.
Results
P2PNet achieves state-of-the-art counting performance and promising localization accuracy.
Takeaways & Limitations
Direct point prediction provides a framework for joint crowd counting and localization aligned with downstream crowd-analysis demands.
Takeaways & Limitations
P2PNet does not explicitly address scale variation, although one-to-one matching may allow implicit scale cues to be learned.
Abstract
from arXiv · showhide
Localizing individuals in crowds is more in accordance with the practical demands of subsequent high-level crowd analysis tasks than simply counting. However, existing localization based methods relying on intermediate representations (\textit{i.e.}, density maps or pseudo boxes) serving as learning targets are counter-intuitive and error-prone. In this paper, we propose a purely point-based framework for joint crowd counting and individual localization. For this framework, instead of merely reporting the absolute counting error at image level, we propose a new metric, called density Normalized Average Precision (nAP), to provide more comprehensive and more precise performance evaluation. Moreover, we design an intuitive solution under this framework, which is called Point to Point Network (P2PNet). P2PNet discards superfluous steps and directly predicts a set of point proposals to represent heads in an image, being consistent with the human annotation results. By thorough analysis, we reveal the key step towards implementing such a novel idea is to assign optimal learning targets for these proposals. Therefore, we propose to conduct this crucial association in an one-to-one matching manner using the Hungarian algorithm. The P2PNet not only significantly surpasses state-of-the-art methods on popular counting benchmarks, but also achieves promising localization accuracy. The codes will be available at: https://github.com/TencentYoutuResearch/CrowdCounting-P2PNet.
1. Introduction
The paper proposes a purely point-based framework for jointly counting and localizing individuals, motivated by the limitations of density maps, pseudo boxes, and existing localization metrics. It introduces nAP and P2PNet, which directly predicts point proposals using one-to-one target matching.
- Motivation: Crowd localization provides finer-grained information for downstream analysis tasks than counting alone.Examples include crowd tracking, activity recognition, abnormality detection, and flow or behavior prediction.
- Limitations of Existing Methods: Existing approaches can be error-prone because density maps omit exact locations, while pseudo bounding boxes and post-processing introduce inaccurate targets or missed detections.The paper specifically highlights difficulties with close instances, scale variation, and NMS-like suppression.
- Proposed Framework: The proposed framework directly uses point annotations as learning targets and outputs points for individual localization, avoiding intermediate density maps and pseudo boxes.Point representation retains high-precision localization while requiring relatively cheaper annotations.
- Evaluation Metric: Density Normalized Average Precision (nAP) evaluates localization and counting errors more comprehensively than image-level absolute counting error alone.The metric is introduced to provide more precise performance evaluation under the point-based framework.
- P2PNet: P2PNet directly predicts point proposals representing heads and uses one-to-one matching to assign optimal learning targets.The method is presented as an intuitive solution consistent with point annotations.
- Results: P2PNet achieves state-of-the-art counting accuracy and promising localization performance.The paper presents these outcomes as evidence for the proposed framework and method.
2. Related Works
Related work is organized around density-map methods, localization-based methods, and localization-aware metrics. The paper argues that these approaches either lack exact localization, rely on error-prone pseudo boxes, or incompletely capture localization and duplicate-prediction errors.
- Density Map based Methods: Density-map methods estimate counts from predicted maps but fail to provide exact individual locations.The paper contrasts this limitation with its direct prediction of individual locations.
- Localization based Methods: Localization-based methods often predict head bounding boxes, but point-only annotations require heuristic pseudo-box estimation that can be error-prone or infeasible.The supplied passage identifies inaccurate boxes as a source of difficulty for training and localization.
- Localization Aware Metrics: Traditional image-level counting metrics ignore spatial variation in estimation errors within an image.Patch-level or pixel-level absolute counting errors were proposed to provide more detailed evaluation.
- Localization Aware Metrics: Mean Localization Error measures average pixel distance, while precision-recall approaches can fail to punish duplicate predictions.The paper describes sequential matching with Average Precision as another existing approach.
- Localization Aware Metrics: The proposed density Normalized Average Precision (nAP) evaluates localization errors and false detections while accounting for density variation.It is designed as a comprehensive metric for both localization and counting errors.
3. Our Work
The paper introduces a purely point-based framework that jointly counts and localizes individuals using point annotations, evaluates both objectives with density Normalized Average Precision, and implements direct point prediction through P2PNet with one-to-one matching.
- The Purely Point-based Framework: The framework directly uses point annotations as learning targets and outputs individual locations, supporting joint crowd counting and localization.It represents individuals by head-center points, optionally with confidence scores.
- The Purely Point-based Framework: Point-based localization avoids labor-intensive head boxes, inaccurate pseudo boxes, and difficult post-processing, especially in highly congested regions.The framework is motivated by the practical needs of downstream crowd-analysis tasks such as tracking and activity recognition.
- Density Normalized Average Precision: Density Normalized Average Precision evaluates localization and counting errors using a density-aware criterion rather than only image-level absolute counting error.Predictions are sorted by confidence and sequentially associated with unmatched ground-truth points to form the precision-recall curve.
- Density Normalized Average Precision: nAP normalizes Euclidean localization distance by the average distance to a point’s k nearest neighbors, with δ controlling the desired localization accuracy.Figure 2 illustrates typical and stricter thresholds using δ=0.5 and δ=0.25, respectively.
- The P2PNet Model: P2PNet predicts point proposals and confidence scores directly, using a VGG16-based upsampling architecture and separate proposal and confidence branches.The approach is designed to remain consistent with point annotations rather than introducing intermediate density maps or pseudo boxes.
- The P2PNet Model: Hungarian one-to-one matching assigns proposals to ground-truth points while preventing duplicate matches that otherwise cause under- or over-estimated counts.Unmatched proposals automatically become negatives, avoiding an additional negative-distance threshold.
4. Experiments
Experiments evaluate P2PNet across crowd-counting datasets, localization metrics, feature configurations, and reference-point layouts. The results show strong counting and localization performance, with resolution benefiting localization and stride 8 balancing varied densities.
- Experimental Setup: Experiments use ShanghaiTech PartA/PartB, UCF CC 50, UCF-QNRF, and NWPU-Crowd, with five-fold cross-validation for UCF CC 50.Training uses random scaling, cropping, and flipping; QNRF and NWPU-Crowd images are capped at dataset-specific maximum sizes.
- Evaluation Metrics: nAP evaluates localization precision across thresholds δ, while MAE and MSE evaluate counting accuracy.nAP0.5 targets typical practical localization requirements, with nAP0.1 and nAP0.25 covering other accuracy levels.
- Model Evaluation: Around 60% overall nAP and generally more than 80% nAP0.5 demonstrate strong localization across datasets.For most datasets, nAP0.5 is nearly 90%, while nAP0.25 remains higher than 55%; nAP0.05 is lower under stricter localization accuracy.
- Model Evaluation: 71.2%/72.9%/69.5% F1-measure/Precision/Recall on NWPU-Crowd is best among published methods with similar backbones.The comparison uses NWPU-Crowd’s scarce box annotations and their localization metrics.
- Model Evaluation: P2PNet achieves strong counting results, including 85.32 MAE on one dataset and a 12.4% MAE reduction over DM-Count on NWPU-Crowd.It also provides exact individual locations, unlike the other methods in the cited comparison.
- Ablation Studies: Reference-point layouts differ only slightly, stride 8 provides a trade-off across densities, and increasing feature-map resolution improves nAP.The finest feature map benefits localization, whereas stride 8 yields better overall performance across varied densities.
5. Conclusion
The paper concludes that a purely point-based framework directly predicts crowd-individual locations while supporting counting. Its nAP metric and P2PNet achieve state-of-the-art counting performance and promising localization accuracy.
- Conclusion: The framework directly predicts individual locations and uses nAP to evaluate localization and counting more comprehensively.P2PNet takes point annotations as supervision and predicts point locations during inference.
- Conclusion: One-to-one matching during target association is identified as P2PNet’s key component and benefits nAP improvement.The framework is presented as conceptually simple while achieving state-of-the-art counting and promising localization accuracy.
1. Counting Evaluation Metrics
The paper adopts MAE and MSE as counting-evaluation metrics. MAE measures the average absolute difference between estimated and ground-truth crowd numbers across test images.
- Counting Metrics: MAE and MSE are adopted as evaluation metrics for crowd-counting accuracy.The definitions follow previous crowd-counting work.
- Counting Metrics: MAE averages the absolute difference between each image’s estimated and ground-truth crowd numbers over N test images.Here, estimated and ground-truth counts are denoted by ẑ_i and z_i, respectively.
2. Discussion on Spatial Scale Problem
P2PNet does not explicitly address scale variation. Because head points are scale-agnostic, one-to-one matching selects one optimal proposal regardless of head scale.
- P2PNet does not explicitly deal with the scale variation problem.The framework is described as compatible with prior scale-variation methods rather than explicitly incorporating them.
3. Hyperparameters Analysis
The number of reference points K is selected from the nearest-neighbour distance distribution of ground-truth points and evaluated for sensitivity.
- Nearly 95% of SHTech PartA head points lie within a 4-pixel nearest-neighbour distance.This observation motivates setting K to 4 on the stride-8 feature map.
- The model’s accuracy sensitivity to the reference-point count K is experimentally analyzed.The analysis includes a comparison with K=1, which still achieves state-of-the-...
4. Localization Performance
P2PNet is evaluated for localization using NWPU-Crowd metrics and nAP comparisons, achieving the best reported performance among comparable methods and outperforming an available baseline.
- P2PNet achieves the best F1 score among published methods with similar computation complexity.
- 33.2%, 45.8% and 8.9% are the baseline nAP0:05:0:05:0:50 results on SHTech PartA, SHTech PartB and QNRF, respectively.P2PNet achieves significantly higher nAP localization performance, especially on QNRF.
5. Visual Results for Qualitative Evaluation
Qualitative examples span sparse, moderately congested, and congested scenes. P2PNet shows localization and counting accuracy across densities, but failures remain on some extreme large heads and gray images.
- P2PNet achieves impressive localization and counting accuracy across sparse, medium, and dense crowd conditions.Figures 7–19 present qualitative examples across these density levels.
- P2PNet may fail on some extreme large heads and gray images such as old photos.The paper notes that adding more relevant training data might alleviate these failures to some extent.
- The qualitative evaluation includes visual results for sparse, moderately congested, and congested scenes.