PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Counting: Comparisons of Density Maps for Crowd Analysis Tasks - Counting, Detection, and Tracking

Di Kang, Zheng Ma, Antoni B. Chan

arXiv:1705.10118v2cs.CV

TL;DR

Low-resolution, heavily occluded crowds challenge object-based analysis, while density maps offer counting with preserved spatial information. The paper compares density-map estimators across counting, detection, and tracking, finding a resolution trade-off between counting efficiency and localization quality. It also introduces metrics for assessing density maps across these tasks.

  • Problem

    Low resolution and severe occlusion make crowded-scene counting, detection, and tracking difficult, while existing CNN density maps often have reduced resolution.

  • Method

    The paper compares density-map estimation methods across counting, detection, and tracking, including full-resolution CNN-pixel and an FCNN adaptation with upsampling and skip connections.

  • Results

    Reduced-resolution maps can perform well for counting, whereas original-resolution maps provide better localization accuracy for detection and tracking than upsampled reduced-resolution maps.

  • Takeaways & Limitations

    Density-map quality metrics can guide methods intended to support both counting and localization tasks.

Abstract

from arXiv · show

For crowded scenes, the accuracy of object-based computer vision methods declines when the images are low-resolution and objects have severe occlusions. Taking counting methods for example, almost all the recent state-of-the-art counting methods bypass explicit detection and adopt regression-based methods to directly count the objects of interest. Among regression-based methods, density map estimation, where the number of objects inside a subregion is the integral of the density map over that subregion, is especially promising because it preserves spatial information, which makes it useful for both counting and localization (detection and tracking). With the power of deep convolutional neural networks (CNNs) the counting performance has improved steadily. The goal of this paper is to evaluate density maps generated by density estimation methods on a variety of crowd analysis tasks, including counting, detection, and tracking. Most existing CNN methods produce density maps with resolution that is smaller than the original images, due to the downsample strides in the convolution/pooling operations. To produce an original-resolution density map, we also evaluate a classical CNN that uses a sliding window regressor to predict the density for every pixel in the image. We also consider a fully convolutional (FCNN) adaptation, with skip connections from lower convolutional layers to compensate for loss in spatial information during upsampling. In our experiments, we found that the lower-resolution density maps sometimes have better counting performance. In contrast, the original-resolution density maps improved localization tasks, such as detection and tracking, compared to bilinear upsampling the lower-resolution density maps. Finally, we also propose several metrics for measuring the quality of a density map, and relate them to experiment results on counting and localization.

I. INTRODUCTION

Crowded, low-resolution scenes make counting, detection, and tracking difficult, motivating density maps that preserve both object counts and spatial arrangement. This paper compares density-map methods across these tasks, including full-resolution CNN variants and quality metrics.

  • Motivation: Low resolution and severe occlusion reduce the performance of object-based counting, detection, and tracking methods in crowded scenes.People may be only a few pixels tall, and occlusion can make analysis difficult even for human experts.
  • Motivation: Density maps bypass explicit detection while preserving spatial information, enabling counting and potential localization of individual objects.Integrating density over a region yields its object count.
  • Study scope: The paper compares CNN-based and classic CNN-based density estimators on counting, detection, and tracking.The comparison includes existing CNN methods, CNN-pixel, and a fully convolutional adaptation.
  • Applications: Density maps support counting by summing predictions within the ROI, detection through clustering or integer programming, and tracking by multiplying KCF responses with density.Figure 1 presents these three uses of the same representation.
  • Study scope: CNN-pixel predicts a density value for every image pixel using a sliding-window regressor, while FCNN adaptations restore resolution through upsampling and skip connections.These methods address the reduced-resolution outputs common in CNN density estimators.
  • Contributions: The paper proposes density-map metrics and relates them to performance on counting and localization tasks.The contributions include metrics intended to indicate density-map quality for localization.

B. Density-based Counting

Density-based counting methods differ in features, losses, prediction granularity, and output resolution. Reduced-resolution maps can be efficient and accurate for counting, but may lose localization detail when upsampled.

  • Density-map representation: Object density maps define the integral over any image subregion as the number of objects in that region.This representation retains spatial information while supporting counting.
  • Traditional methods: Traditional density-map methods use features such as raw images, temporal differences, background subtraction, filter responses, and random-forest representations.Feature choices vary across methods and may combine multiple information sources.
  • Deep methods: Deep methods include patch-wise CNNs, multi-scale Hydra CNN and MCNN architectures, CNN-boost residual prediction, and fully convolutional density estimation.MCNN uses columns with different receptive-field settings to capture objects at different scales.
  • Loss and prediction: Density estimation methods vary in whether their loss and prediction operate pixel-wise or region-wise.MESA is a region-based loss designed to preserve counting performance across subregions, whereas many other methods use pixel-wise squared error.
  • Output resolution: Patch-wise and image-wise CNN methods usually produce reduced-resolution maps because of downsampling or the cost of wide fully connected layers.Reduced resolution can speed prediction and still provide good counting performance.
  • Output resolution: Upsampling reduced-resolution maps can create overly spread-out density that localizes individual objects poorly, making original-resolution maps more suitable for detection.Overlapping prediction and averaging can also produce overly smooth maps.

C. Detection and Tracking with Object Density Maps

Density maps support localization because they retain spatial structure, while full-resolution CNN designs provide detailed maps for detecting and tracking crowded objects. The pixel-wise CNN combines density regression with an auxiliary patch-count classification task, and CNN-derived maps are described as less noisy and better localized than handcrafted-feature methods.

  • C. Detection and Tracking with Object Density Maps: Density maps preserve spatial information, enabling detection and tracking in addition to counting.Their use for localization builds on retaining where objects are distributed rather than only predicting a global count.
  • C. Detection and Tracking with Object Density Maps: CNN-pixel predicts the density at each patch center and reconstructs a full-resolution map by sliding the patch across the image.The network uses a 33×33 input patch, with the patch size chosen to resemble the largest person in the image.
  • C. Detection and Tracking with Object Density Maps: CNN-derived density maps are less noisy and better localized around objects than maps based on handcrafted features.The paper attributes this comparison to CNNs’ learned feature representations.
  • C. Detection and Tracking with Object Density Maps: CNN-pixel training combines center-pixel density regression with an auxiliary classification task that predicts the people count in the image patch.Both tasks share CNN feature-extraction layers; the auxiliary task is intended to guide feature learning and reduce sensitivity to initialization and learning rate.
  • C. Detection and Tracking with Object Density Maps: The auxiliary classification loss is used only during training, while test-time counting sums the predicted density map.The combined objective uses weights λ1 = 100 and λ2 = 1 in the reported implementation.

B. Fully Convolutional Architecture

The fully convolutional adaptation predicts an entire density map efficiently by reusing computations from overlapping regions. Upsampling restores original resolution, while skip branches add lower-level features to compensate for spatial information lost through convolution and pooling strides.

  • B. Fully Convolutional Architecture: Fully convolutional networks efficiently produce dense maps by reusing computations from overlapping image patches.This whole-image prediction strategy avoids separately evaluating every overlapping patch.
  • B. Fully Convolutional Architecture: FCNN-skip adapts CNN-pixel with skip branches that add lower-layer features to upsampled representations.The skip branches are designed to compensate for spatial information lost during stride operations.
  • B. Fully Convolutional Architecture: Two pooling operations reduce density-map resolution by 4, followed by two upsampling layers that restore the original resolution.Each upsampling layer replicates pixels into a 2×2 region and applies a trainable 3×3 convolution.
  • B. Fully Convolutional Architecture: FCNN training uses both pixel-wise reconstruction loss and patch-wise count loss.The pixel-wise term targets per-pixel reproduction, while the count term tunes the network toward counting.
  • B. Fully Convolutional Architecture: At prediction time, FCNN-skip processes the whole image and outputs a density map at the same resolution without block artifacts.The network is initialized from a trained CNN-pixel model, so no auxiliary task is needed for FCNN training.

C. Detection from Density Maps

Detection methods can recover object locations from density maps using sliding-window counts, clustering, or local peaks. The paper also weights GMM clustering by density values so cluster centers move toward high-density regions, while density-map fusion helps KCF tracking avoid drift.

  • C. Detection from Density Maps: Density-map detection can begin by computing object counts in sliding windows and recovering locations as a deconvolution problem.The earlier approach solves recovery with two-dimensional integer programming because object counts at locations are nonnegative integers.
  • C. Detection from Density Maps: Fusing the KCF response map with the crowd density map downweights an erroneous response peak and prevents tracker drift.The example contrasts the drifting KCF result with the corrected fused result.
  • C. Detection from Density Maps: Baseline localization methods include non-maximum suppression, k-means clustering, and GMM clustering on thresholded density maps.K-means and GMM use crowd-blob shape while ignoring density values.
  • C. Detection from Density Maps: CNN-pixel’s well-defined peaks motivate density-weighted GMM, which shifts cluster centers toward high-density regions likely to contain objects.The weighting is intended to better locate local peaks.
  • C. Detection from Density Maps: The paper evaluates density-map quality through attributes intended to relate high-resolution map reproduction to detection and tracking performance.The quality analysis includes scatter plots comparing ground-truth and predicted density values and experiments across several map methods.

A. Per-pixel Reproduction

Per-pixel reproduction assesses how closely predicted density values match the ground-truth map, while compactness, localization, and temporal smoothness evaluate suitability for spatial and tracking tasks.

  • Per-pixel reproduction: MCNN, CNN-pixel, and FCNN-skip show the strongest correlation between predicted and ground-truth per-pixel densities.Ridge regression over-predicts, CNN-patch under-predicts, and MESA is less concentrated around the diagonal because it optimizes regional count error.
  • Compactness and localization: BBDR measures density compactness, while BBMAE measures localization accuracy inside perspective-scaled bounding boxes.Higher BBDR indicates greater compactness, whereas lower BBMAE indicates better localization.
  • Compactness and localization: CNN-pixel achieves the best localization and ground-truth-like compactness; FCNN-skip localizes well but spreads density during upsampling.MCNN-up improves compactness and localization over MCNN, while ridge regression is compact but poorly localized because local-mode centroids shift.
  • Temporal smoothness: CNN-based density maps are temporally smoother than traditional methods, but CNN-patch is smoothest because its density is more spread out.Greater temporal smoothness does not by itself ensure useful localization or tracking.
  • Temporal smoothness: CNN-pixel provides the lowest EDD among tested detection settings, indicating the most stable detected points over time.CNN-pixel and FCNN-skip have similar ED with IntProg, while both obtain the lowest ED with GMM-weighted.

D. Summary

The paper compares density estimators across counting, detection, and tracking, emphasizing how output resolution affects density-map quality and downstream localization. Full-resolution CNN-pixel generally provides the strongest localization-related metrics, while FCNN alternatives improve efficiency with some quality loss.

  • D. Summary: CNN-pixel performs best on compactness, localization, and temporal stability metrics.Reduced-resolution maps requiring fixed or learned upsampling have worse quality metrics because downsampling obscures object positions.
  • D. Summary: The experiments compare recent CNN and traditional density estimators on crowd counting, detection, and tracking tasks.The evaluated methods include MESA, ridge regression, regression forests, CNN-patch, MCNN, Hydra CNN, CNN-boost, CNN-pixel, and FCNN-skip.
  • D. Summary: FCNN-skip retains the smoothness of pixel-wise prediction while reducing redundant inference computation.Its architecture adapts CNN-pixel into a fully convolutional network with upsampling and skip branches.
  • D. Summary: CNN-pixel predicts full-resolution density values pixel by pixel, whereas FCNN-skip predicts whole images using upsampling and skip connections.FCNN processing reuses computations from overlapping regions and is more efficient at inference than sliding-window CNN-pixel prediction.

1) Datasets:

The study evaluates density estimation on pedestrian and vehicle datasets spanning low-resolution surveillance, diverse scenes, extreme crowd sizes, and traffic congestion. On UCSD, CNN-based methods generally reduce counting error, while full-resolution CNN-pixel is especially strong for localization-oriented density quality.

  • 1) Datasets:: The evaluation uses UCSD, WorldExpo’10, UCF CC 50, and TRANCOS, covering pedestrians and vehicles across varied crowd conditions.The datasets include low-resolution surveillance, novel scenes, extreme crowd sizes, and traffic congestion with partial occlusion.
  • 1) Datasets:: UCSD is a 238×158 low-resolution surveillance dataset with 2,000 frames, 49,885 pedestrians, perspective change, and heavy occlusion.Its protocols test both training-data robustness and generalization across crowd levels.
  • 1) Datasets:: WorldExpo’10 contains 3,980 annotated images, 199,923 pedestrians, and 108 scenes, with training on 103 scenes and testing on five novel scenes.Perspective-dependent density templates generate ground-truth maps, and fractional ROI counts handle people near ROI boundaries.
  • 2) UCSD pedestrian dataset:: On full-training UCSD, MCNN obtains MAE 1.07, slightly below CNN-boost at 1.10 and CNN-pixel at 1.12.FCNN-skip reaches MAE 1.22 but predicts much faster than CNN-pixel: 16 ms versus 4.6 sec per frame.
  • 2) UCSD pedestrian dataset:: Across UCSD’s four reduced-training splits, CNN-pixel and ridge regression achieve the lowest average MAE.MCNN and FCNN-skip are more computationally efficient but perform somewhat worse, particularly on the upscale split.
  • 2) UCSD pedestrian dataset:: Using the full training set lowers CNN-pixel and MCNN MAE relative to the maximal split, which uses only one-fifth as much training data.The paper also presents CNN-pixel density-map examples and cross-method visual comparisons.

3) WorldExpo’10 dataset:

On WorldExpo’10, MCNN achieved the lowest average counting error, while performance varied across scenes and challenging crowd configurations exposed prediction failures.

  • Counting performance: MCNN had the lowest average error on WorldExpo’10, while CNN-pixel, FCNN-skip, and CNN-patch without fine-tuning performed similarly.The table reports MAE values for the compared methods, including MCNN at 377.6, CNN-pixel at 406.2, and FCNN-skip at 431.6.
  • Scene variation: Scenes 2, 3, and 4 produced larger errors than Scenes 1 and 5 because they contained more people on average.Only 12% of training frames contained large crowds exceeding 80 people.
  • Failure cases: On Scene 3, CNN-pixel under-predicted density where partially occluded people appeared on a roof inside the region of interest.The ground truth assigns non-zero density to the roof, whereas CNN-pixel predicts zero there because it sees no human parts.
  • Failure cases: On Scene 4, a region-of-interest boundary crossing a background crowd caused density outside the ROI to confound CNN-pixel predictions and increase errors.The human-shaped ground truth places substantial density outside the ROI, while CNN-pixel predictions tend to be larger.

5) TRANCOS dataset:

On TRANCOS, CNN methods substantially outperformed traditional-feature methods, and FCNN-skip achieved the best reported errors across the listed GAME levels.

  • Evaluation: GAME evaluates estimated-versus-ground-truth count error within grid subregions, with the number of subregions determined by level L.At L = 0, GAME(0) equals MAE.
  • Evaluation: FCNN-skip had the lowest MAE, GAME(1), and GAME(2) among the evaluated methods on TRANCOS.GAME measures counting error within grid subregions; GAME(0) is equivalent to MAE.
  • Comparison: CNN methods surpassed methods using traditional features by a large margin on the TRANCOS evaluation.

2) Results:

Detection performance depended on preserving accurate, compact, and isolated density peaks: CNN-pixel and MESA performed best with IntProg, while upsampling affected localization differently across methods.

  • Detection: Using IntProg with CNN-pixel or MESA density maps yielded the best detection F1 score on UCSD.CNN-pixel provided higher precision but lower recall than MESA, reflecting a compactness-localization trade-off.
  • Detection: FCNN-skip detection was worse than CNN-pixel because upsampling produced less compact and less isolated density maps.Skip connections compensated for some spatial-information loss but did not prevent degradation in detection accuracy.
  • Detection: MCNN-up improved F1 score over MCNN across detection methods, although counting performance decreased slightly from 1.32 to 1.37 MAE.Learned upsampling improved compactness and localization compared with MCNN, unlike bicubic upsampling.
  • Density-map properties: RR produced highly compact maps but shifted density-blob centers, worsening localization and detection with IntProg and GMM-weighted.
  • Density-map properties: CNN-patch detection was worse because reduced resolution and averaging overlapping predictions smoothed local peaks, making individual localization harder.
  • Density-map properties: Accurate monotonic peaks and compact, isolated responses were identified as important for the tested detection methods.With higher-quality maps, GMM-weighted performed similarly to IntProg; IntProg still showed a gap between ground-truth and predicted maps, with F1 97.96 versus 92.89.

2) Results:

Across counting, detection, and tracking, original-resolution density maps preserved localization better than upsampled reduced-resolution maps, while training choices shaped density-map structure and counting accuracy.

  • Tracking: CNN-pixel fusion produced the largest tracking-precision increase, reaching P@4 of 0.713 versus 0.692 for RR.The result was associated with better localization and temporally more stable detections; ground-truth fusion provided the upper bound.
  • Summary: Reduced-resolution maps such as MCNN could count accurately, whereas original-resolution maps generally improved localization for detection and tracking.Per-pixel original-resolution maps such as CNN-pixel had higher compactness and localization fidelity than maps upsampled from reduced resolution.
  • Summary: Downsampling obfuscated people’s true positions, and learned upsampling with skip connections only partially compensated for the lost spatial information.
  • Training and architecture variations: Increasing CNN-pixel depth produced no gain with four convolution layers and slightly reduced performance with six layers, while ResNet and DenseNet variants did not improve counting.The six-layer variant achieved MAE 1.38, and the ResNet version achieved MAE 1.26.
  • Training and architecture variations: Using 256 rather than 512 neurons in CNN-pixel’s first fully connected layer worsened performance to MAE 1.38.The authors suggest more neurons may capture appearance variations associated with the same density value.
  • Training and architecture variations: Removing FCNN-skip’s count loss increased MAE to 1.41 because count loss targets systematic counting errors not well reflected by pixel-wise loss.
  • Training and architecture variations: Training FCNN with only count loss spread density maps and yielded MAE 1.82, indicating that pixel-wise loss maintained density-map structure.
  • Training and architecture variations: CNN-pixel achieved MAE 1.26 with λ2 = 1, while λ2 values of 0, 10, and 100 produced 1.41, 1.44, and 1.47.λ2 = 0.1 performed similarly to λ2 = 1.

VI. CONCLUSION

Reduced-resolution density maps perform well for counting, but full-resolution dense predictions provide the highest-quality localization maps, despite greater computational complexity. Proposed density-map metrics help explain these task differences and guide future designs.

  • Reduced-resolution fully convolutional density maps perform well at counting, whereas full-resolution CNN-pixel maps provide the highest-quality localization for detection and tracking.Upsampling and skip connections could not completely recover the spatial-resolution loss in reduced-resolution maps.
  • Full-resolution dense prediction slightly degrades counting performance compared with reduced-resolution maps.
  • Dense pixel prediction has higher computational complexity than fully convolutional networks.
  • The proposed metrics measure density-map aspects that explain why maps with similar counting accuracy can differ on detection and tracking.These metrics are intended to guide future density-map designs for both counting and localization.
Loading 1705.10118v2…