PaperlayerView exact revision on arXiv

Source-linked AI summary

Crowd Counting with Decomposed Uncertainty

Min-hwan Oh, Peder A. Olsen, Karthikeyan Natesan Ramamurthy

arXiv:1903.07427v3cs.CVcs.LG

TL;DR

CNN crowd-counting methods often provide point estimates without quantifying uncertainty, a concern for high-stakes crowd analysis. The paper proposes a scalable bootstrap-ensemble framework that decomposes uncertainty and reports state-of-the-art performance across considered benchmarks, with calibrated and meaningful uncertainty estimates.

Abstract

from arXiv · show

Research in neural networks in the field of computer vision has achieved remarkable accuracy for point estimation. However, the uncertainty in the estimation is rarely addressed. Uncertainty quantification accompanied by point estimation can lead to a more informed decision, and even improve the prediction quality. In this work, we focus on uncertainty estimation in the domain of crowd counting. With increasing occurrences of heavily crowded events such as political rallies, protests, concerts, etc., automated crowd analysis is becoming an increasingly crucial task. The stakes can be very high in many of these real-world applications. We propose a scalable neural network framework with quantification of decomposed uncertainty using a bootstrap ensemble. We demonstrate that the proposed uncertainty quantification method provides additional insight to the crowd counting problem and is simple to implement. We also show that our proposed method exhibits the state of the art performances in many benchmark crowd counting datasets.

Abstract

The paper addresses the lack of uncertainty estimates in CNN crowd counting and proposes a scalable bootstrap-ensemble framework that decomposes uncertainty. It reports state-of-the-art performance on multiple crowd-counting benchmarks.

Abstract

  • CNN crowd-counting methods provide point estimates without addressing uncertainty in predictions.The authors note that confidence information can support decisions such as referring uncertain images or regions for human validation.Source: Introduction · p. 1
  • The framework uses a bootstrap ensemble to quantify uncertainty from both the model and input data in crowd counting.The authors describe it as simple to implement and scalable.Source: Introduction · p. 1
  • The authors state that this is the first work, to their knowledge, to address uncertainty quantification for neural-network crowd-counting predictions.They report that the method produces accurate estimated uncertainty.Source: Introduction · p. 1
  • The authors report state-of-the-art-level performance on multiple crowd-counting benchmark datasets.They also describe the architecture as more computationally efficient than previously known state-of-the-art methods.Source: Introduction · p. 2
  • The uncertainty framework is described as generic, independent of the underlying network architecture, and adaptable to other architectures.Source: Introduction · p. 2

Related Work

Prior crowd-counting methods include detection, regression, and density estimation, with CNN density methods achieving strong performance. The section identifies uncertainty assessment in predictive outputs as a comparatively neglected issue.

Related Work

  • Detection-based counting is challenged by occlusion and large differences in object scale, making it difficult to use in dense crowd scenes.Source: Related Work · p. 2
  • Regression-based methods map image features to scalar counts, bypassing explicit detection, but mostly ignore spatial information in crowd images.Source: Related Work · p. 2
  • Density maps preserve count and spatial distribution while avoiding the need to detect every heavily occluded object; earlier methods relied on handcrafted features.Source: Related Work · p. 2
  • CNN density-based methods have outperformed traditional handcrafted-feature approaches, with later work addressing perspective and context through varied architectures.Source: Related Work · p. 2
  • Despite strong crowd-counting performance, less attention has been paid to uncertainty in predictions, and point estimates alone may be spuriously confident.Source: Related Work · p. 2

Uncertainty in Neural Networks

The section frames bootstrap ensembles as a non-parametric alternative for uncertainty estimation, while distinguishing model uncertainty from noise inherent in the data. It also motivates recalibration because uncertainty estimates may not match their stated confidence levels.

Uncertainty in Neural Networks

Bootstrap ensemble

  • A bootstrap ensemble trains K models on independently resampled datasets and samples a model to approximate a distribution over functions.The approach scales to high-dimensional spaces using point estimates, but its computational load grows linearly with the number of base models.Source: Bootstrap ensemble · pp. 2–3

Measures of uncertainty

  • Epistemic uncertainty reflects limited model knowledge, whereas aleatoric uncertainty reflects genuine data noise that can persist despite more training data.The paper incorporates both sources for crowd counting, where occlusion and perspective distortions can create inherent noise.Source: Measures of uncertainty · p. 3

Calibration of Predictive Uncertainty

  • Bootstrap uncertainty estimates may be miscalibrated, so the paper adopts a regression recalibration procedure that is guaranteed to yield calibrated estimates given enough data.The guarantee is attributed to the cited recalibration procedure applied to Bayesian models.Source: Calibration of Predictive Uncertainty · p. 3

Proposed Method

DUBNet uses shared network layers with K bootstrap heads to estimate epistemic uncertainty and an additional output for input-dependent noise. The method trains these outputs jointly and recalibrates count uncertainty using validation data.

Single network with K output heads

  • A shared feature network branches into K bootstrap heads, reducing the computational burden of maintaining several independent networks at the cost of lower head diversity.The heads can be trained in a single forward/backward pass.Source: Single network with K output heads · p. 3

Capturing epistemic uncertainty

  • For epistemic uncertainty, training samples one bootstrap head per iteration, while test-time predictions combine all K heads to estimate the predictive mean and variance.The method places a prior over shared and head-specific weights and uses a Gaussian pixel-wise likelihood.Source: Capturing epistemic uncertainty · pp. 3–4

Incorporating aleatoric uncertainty

  • The network predicts both a density map and input-dependent noise variance, allowing the loss to balance squared residuals against an aleatoric-uncertainty penalty.Predicting log variance instead of variance supports numerical stability while representing a positive variance.Source: Incorporating aleatoric uncertainty · p. 4

Network architecture

  • DUBNet combines a ResNet-50 feature-extraction front end, a dilated-convolution back end, and bootstrap output heads with an extra log-variance output.The paper reports 3.8B FLOPs for ResNet-50 versus 15.3B FLOPs for VGG-16.Source: Network architecture · p. 4, Figure 1 · p. 5

Training procedure

  • Training samples a bootstrap head uniformly for each image, initializes the first 10 convolutional layers from pretrained ResNet-50, and uses Adam with a learning rate of 10^-5.Source: Training procedure · p. 4

Recalibration of Predictive Uncertainty

  • Recalibration fits an isotonic regression model to standardized count residuals and their empirical quantiles from validation data, then uses its quantiles to form test-time confidence bounds.A bound is computed as the predicted count plus the predictive count standard deviation multiplied by the recalibrated quantile.Source: Recalibration of Predictive Uncertainty · p. 5

Experiments

Experiments evaluate DUBNet and its uncertainty estimates on crowd-counting benchmarks, with ablations testing the bootstrap extension and its adaptability to other architectures. The method ranks among the top two on almost all benchmark datasets considered, and its estimated uncertainty is reported as calibrated and meaningful.

Experiments

Evaluation metrics

  • Crowd-count estimation error is evaluated using mean absolute error (MAE) and root mean squared error (RMSE).The test-time estimate uses the predictive mean over K bootstrap outputs.Source: Evaluation metrics · p. 5

Ablation study

  • The ablation results support applying the framework's aleatoric and epistemic uncertainty components to CSRNet and MCNN as well as DUBNet.The authors report significant performance contributions on evaluation data and describe the framework as applicable to other architectures.Source: Table 1 · p. 5, Ablation study · p. 6

Performance comparisons

  • UCF-CC 50 is evaluated with five-fold cross-validation because the dataset provides no training/test split and contains only 50 images.UCF-QNRF training images exceeding 3 million pixels are down-sampled to address GPU memory issues.Source: Performance comparisons · p. 6

Results

  • DUBNet is among the top two performers on almost all benchmark datasets considered.The estimation-error table covers ShanghaiTech A/B, UCF-CC 50, and UCF-QNRF.Source: Results · p. 6

Estimated uncertainty validation

  • Estimated uncertainty is evaluated for calibration by checking whether its estimated p-quantile confidence interval contains the true outcome p fraction of the time; test-data ground truths fall in the 90% CI, suggesting accuracy.Source: Estimated uncertainty validation · p. 6

Discussion on estimated uncertainty

  • Epistemic uncertainty is generally higher in dense regions, while aleatoric uncertainty is more prominent in areas with image noise or occlusion.The discussion notes low aleatoric uncertainty in a crowded stadium image with few occlusions or perspective issues.Source: Discussion on estimated uncertainty · p. 6

Conclusion

The paper concludes that DUBNet combines shared convolutional layers and bootstrap ensembles to estimate decomposed uncertainty in crowd counting. It reports state-of-the-art-level benchmark performance with calibrated, meaningful uncertainty estimates, while noting that limited UCF-CC 50 data prevents uncertainty recalibration.

Conclusion

  • The framework combines shared convolutional layers with bootstrap ensembles to decompose crowd-count uncertainty into epistemic and aleatoric components.The conclusion describes the approach as scalable and independent of architecture choices.Source: Conclusion · p. 6
  • Uncertainty recalibration is not performed on UCF-CC 50 because the dataset is too limited in size for that procedure.Source: Conclusion · p. 6
  • DUBNet achieves state-of-the-art-level performance on the benchmark datasets considered and produces calibrated, meaningful uncertainty estimates.The conclusion's estimation-error and calibration tables cover benchmark performance and uncertainty calibration, respectively.Source: Conclusion · p. 7, Table 2 · p. 7, Table 3 · p. 7

CNN giga-pixel imagery for the 2017 U.S. Presidential inauguration

The authors apply DUBNet to unlabeled, ultra-high-resolution inauguration imagery, where a single fixed zoom level does not work best across all views. They adapt zoom by image region and report counts with uncertainty, while limiting claims to people visible in the images.

CNN giga-pixel imagery for the 2017 U.S. Presidential inauguration

  • The inauguration imagery comprises six viewing directions, each with 15,625 patches of 500 × 500 pixels, or approximately 3.9 giga-pixels.The zoomed-out versions have seven levels but are stitched from the original high-resolution images, not additional photos.Source: CNN giga-pixel imagery for the 2017… · p. 10
  • Because the imagery has no ground-truth counts, predictions concern only people visible in the images, not total inauguration attendance or unseen areas.A white temporary structure blocks a significant portion of the crowd in one view.Source: CNN giga-pixel imagery for the 2017… · p. 10
  • No single zoom level works best across all viewing directions; Table 4 compares fixed zoom levels, while Table 5 reports aggregated partition counts with confidence intervals.The authors observe that counts rise with zoom level and that zoom level 7 overestimates while level 3 seems too low in the example discussed.Source: CNN giga-pixel imagery for the 2017… · p. 14, Table 4 · p. 14, Table 5 · p. 14
  • The method adaptively selects zoom by region, requiring predicted counts to reach at least 20 people before choosing the zoom with the lowest predictive variance.Regions that miss the threshold are zoomed out; the process iteratively partitions the image and reports counts and density maps per partition.Source: CNN giga-pixel imagery for the 2017… · p. 14
  • Most sample partitions produce plausible density maps and counts, but the first partition in Figure 4 is especially challenging and may be underestimated.Table 5 reports aggregated counts and corresponding confidence intervals for each viewing direction.Source: CNN giga-pixel imagery for the 2017… · p. 14, Table 5 · p. 14

Discussions on Bayesian neural network

Bayesian neural networks treat network weights as random variables and infer a posterior, but computing it is challenging and often costly. The discussion motivates bootstrap ensembles after Monte Carlo dropout did not produce satisfying confidence intervals for crowd counting.

Discussions on Bayesian neural network

  • The authors proceed without ground-truth labels, while noting that practitioners may need to provide some labeled data for accurate confidence intervals.The footnote presents this as a possible minimal labeled-data requirement.Source: Discussions on Bayesian neural network · p. 14
  • Bayesian neural networks treat network weights as random variables with a prior, yielding a posterior distribution over weights given the data.The discussion assumes a collection of i.i.d. image and density-map realizations.Source: Discussions on Bayesian neural network · pp. 14–15
  • Computing the posterior requires integration over the parameter space, often without a closed form, and Bayesian approaches can be computationally expensive.The discussion describes Laplace approximation, Hamiltonian Monte Carlo, and variational inference as approaches to posterior approximation.Source: Discussions on Bayesian neural network · p. 15
  • Monte Carlo dropout offers a relatively simple uncertainty-estimation approach, but it did not produce satisfying confidence intervals for this crowd-counting problem.This motivates the paper's consideration of a non-parametric bootstrap of functions.Source: Discussions on Bayesian neural network · p. 15

Ground truth generation

Ground-truth density maps are generated by blurring head annotations with normalized Gaussian kernels. The kernel spread adapts to local crowd density using distances to nearby annotated objects.

Ground truth generation

Comparison on variability

DUBNet has slightly lower pre-recalibration predictive variance than a full bootstrap ensemble, but the difference is not large; after recalibration, their uncertainties are almost identical.

Comparison on variability

Note on Inference Runtime

The bootstrap extension adds little inference time: the measured increase was 2% on CPU and 0.5% on GPU. Training adds no computation per epoch under the reported sampling procedure.

Note on Inference Runtime

Loading 1903.07427v3…