PaperlayerView exact revision on arXiv

Source-linked AI summary

$CrowdDiff$: Multi-hypothesis Crowd Density Estimation using Diffusion Models

Yasiru Ranasinghe, Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, Vishal M. Patel

arXiv:2303.12790v3cs.CV

TL;DR

Crowd counting with density maps is limited by background-noise accumulation and density loss from broad kernels. CrowdDiff uses conditional denoising diffusion to generate narrow-kernel density maps, thresholds and fuses multiple stochastic realizations, and uses an auxiliary regression branch during training. The method reports state-of-the-art results on public crowd-counting datasets, while more realizations improve performance at the cost of inference time.

  • Problem

    Density-based counting accumulates background noise and loses density because broad kernels interfere in congested regions, while narrow-kernel regression performs poorly.

    Source: §1 · pp. 1–2

  • Method

    CrowdDiff generates density maps through conditional reverse diffusion, thresholds and fuses multiple realizations, and uses an auxiliary regression branch only during training.

    Source: §1 · p. 2

  • Results

    CrowdDiff surpasses state-of-the-art crowd-counting methods on public datasets.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    Narrow-kernel diffusion generation supports threshold-based counting and stochastic multi-realization fusion for crowd density estimation.

    Source: §1 · p. 2, §6 · p. 7

    Increasing the number of realizations raises inference time, and the gain from four to eight realizations is insignificant while inference time doubles.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Crowd counting is a fundamental problem in crowd analysis which is typically accomplished by estimating a crowd density map and summing over the density values. However, this approach suffers from background noise accumulation and loss of density due to the use of broad Gaussian kernels to create the ground truth density maps. This issue can be overcome by narrowing the Gaussian kernel. However, existing approaches perform poorly when trained with ground truth density maps with broad kernels. To deal with this limitation, we propose using conditional diffusion models to predict density maps, as diffusion models show high fidelity to training data during generation. With that, we present $CrowdDiff$ that generates the crowd density map as a reverse diffusion process. Furthermore, as the intermediate time steps of the diffusion process are noisy, we incorporate a regression branch for direct crowd estimation only during training to improve the feature learning. In addition, owing to the stochastic nature of the diffusion model, we introduce producing multiple density maps to improve the counting performance contrary to the existing crowd counting pipelines. We conduct extensive experiments on publicly available datasets to validate the effectiveness of our method. $CrowdDiff$ outperforms existing state-of-the-art crowd counting methods on several public crowd analysis benchmarks with significant improvements.

1. Introduction

CrowdDiff addresses background-noise accumulation, density loss, and narrow-kernel limitations in crowd counting by generating density maps with diffusion models and exploiting multiple stochastic realizations. It uses thresholded density kernels, fusion, and an auxiliary training-only counting branch, and reports state-of-the-art results on public datasets.

Problem

  • Broad density kernels accumulate background noise and lose density in congested regions, while narrow-kernel regression methods have been ineffective.Density-based counting sums pixel values, making background noise affect the final count; narrow kernels can reduce interference but remain difficult for regression networks.Source: §1 · p. 1

Motivation

  • Diffusion-based generation enables narrow kernels that preserve density distributions and reproduce narrow density structures in dense regions where recent methods fail.The proposed method uses diffusion models to learn density-map value distributions with greater fidelity to training data.Source: §1 · p. 2

Method

  • The method thresholds predicted density maps to count observable kernels, avoiding density summation and reducing the effect of background noise.Multiple thresholded crowd maps are fused into one crowd map for counting.Source: §1 · pp. 1–2
  • CrowdDiff produces and fuses multiple density-map realizations, while an auxiliary regression branch is used only during training to improve feature learning.The multiple-realization mechanism exploits diffusion-model stochasticity; the regression branch is discarded during inference.Source: §1 · p. 2

Contributions

  • CrowdDiff generates crowd density maps with denoising diffusion models and reports state-of-the-art performance on public crowd-counting datasets.The method is presented as the first crowd-counting study using diffusion models.Source: §1 · p. 2

2. Background and Related Work

Prior crowd-counting methods use direct count prediction, density estimation, detection, or localization, with density methods vulnerable to background noise and density loss from broad kernels. CrowdDiff instead treats density-map prediction as a conditional diffusion task, predicting noise through reverse diffusion and using hybrid training objectives.

2.1. Crowd counting

  • Density-based crowd counting suffers from background noise and density loss in congested regions because broad Gaussian kernels blur neighboring densities.Narrow Gaussian kernels are ineffective with conventional regression networks, motivating a generative formulation.Source: §2.1 · p. 2

2.2. Diffusion models for crowd density generation

  • Diffusion models add Gaussian noise in a forward Markov process and remove it through a learned reverse process.The diffusion formulation transforms a clean sample x_0 into noisy samples x_t across a scheduled sequence of timesteps.Source: §2.2 · p. 2
  • CrowdDiff models crowd density maps as diffusion samples and predicts noise conditioned on the crowd image before applying reverse diffusion to recover the density map.The density map is represented as x_0, with spatial dimensions H and W.Source: §2.2 · p. 2
  • The denoising network is optimized with a hybrid loss combining weighted noise-prediction error and a variational lower-bound term.The weighting scheme uses signal-to-noise ratio and hyperparameters k and γ to emphasize coarse-feature learning at lower-SNR stages.Source: §2.2 · p. 2
  • The overall pipeline generates multiple density maps, thresholds them into crowd maps, fuses the maps, and discards the parallel counting branch during inference.The counting branch uses denoising U-Net encoder-decoder features and is trained only as an auxiliary component.Source: Figure 2 · p. 3, §2.2 · p. 3

3. CrowdDiff

CrowdDiff uses narrow-kernel density maps, auxiliary count regression, and stochastic fusion of multiple generated maps to improve crowd counting.

3. CrowdDiff

  • The overall CrowdDiff pipeline trains a denoising U-Net on noisy narrow-kernel maps and reconstructs density maps through reverse diffusion.The section organizes the method around kernel selection, auxiliary counting, and multi-realization fusion.Source: §3 · p. 3, §3.1 · p. 3, §3.3 · p. 4

3.1. Narrow kernel

  • Narrow Gaussian kernels reduce density interference and preserve pixel values, enabling counts from thresholded observable kernels instead of summed density values.Broad kernels create image-dependent distributions and can clip values in congested scenes; thresholding narrow kernels yields head locations without local-maxima hyperparameter tuning.Source: §3.1 · p. 3

3.2. Joint learning of counting

  • CrowdDiff adds a training-only regression branch that pools intermediate denoising features and estimates counts across noise levels.The count loss is an L1 objective weighted by timestep, while the overall objective combines denoising and counting losses.Source: §3.2 · pp. 3–4

3.3. Stochastic crowd map fusion

  • Multiple stochastic density-map realizations are fused by thresholding them into dot maps, ordering by cumulative SSIM, and adding nonredundant locations.Fusion does not require ground-truth locations; it starts with the least similar map and uses rejection radii to remove repeated points.Source: §3.3 · p. 4

4. Experimental Details

The implementation uses a 1,000-step diffusion process with DDIM inference and evaluates CrowdDiff on six public crowd-counting datasets using MAE and MSE.

4. Experimental Details

  • CrowdDiff is evaluated on six public datasets using MAE and MSE, with DDIM sampling during inference.The datasets are JHU-Crowd++, ShanghaiTech A, ShanghaiTech B, UCF-CC-50, UCF-QNRF, and NWPU-Crowd.Source: §4 · p. 5
  • Training uses 1,000 diffusion timesteps, a linear noise schedule, ImageNet-pretrained initialization, and 2 × 10^5 iterations on 256 × 256 images.Optimization uses AdamW with a fixed learning rate of 1 × 10^-4 and linear warm-up over 5 × 10^3 steps.Source: §4 · p. 5
  • Crowd-map fusion uses β = 0.85, at most four nearest neighbors, and a search radius equal to 0.05 of the minimum image dimension.The counting-loss weight is λ_count = 5 × 10^-3, while γ = 0.5 and k = 1 define the SNR-based weighting factors.Source: §4 · p. 5

5. Results

CrowdDiff achieves state-of-the-art crowd-counting performance on public benchmarks, using diffusion-based density generation, narrow kernels, thresholding, and multi-realization fusion. Ablations show that these design choices improve density fidelity, counting, and robustness, while increasing inference time.

5.1. Crowd counting performance

  • CrowdDiff achieves state-of-the-art crowd-counting results on public datasets, including JHU-Crowd++, UCF-QNRF, and NWPU-Crowd.The reported gains are attributed to narrow kernels, reduced background-noise accumulation, and accurate diffusion-generated density maps.Source: §5.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6
  • The proposed pipeline accurately counts dense regions and identifies head locations without localization heuristics, unlike the compared density-based and localization-based methods.Source: §5.1 · p. 5

5.2. Ablation study

  • Diffusion models produce higher-quality narrow-kernel density maps and more accurate counts than the GAN-based ACSCP method.Without narrow kernels, GAN-based density summation reintroduces noise accumulation and density loss.Source: §5.2 · p. 5, Figure 6 · p. 7
  • Combining stochastic realizations requires fusion because shifted or newly generated dots can otherwise create duplicate counts, while narrow kernels preserve novel information.The counting branch improves average counting error and reduces variation across realizations by supporting feature learning at noisy diffusion steps.Source: §5.2 · pp. 5–6, Figure 7 · p. 7
  • Ascend-SSIM fusion, which combines locally dissimilar realizations first, improves counting performance over Random and Descend-SSIM fusion.Source: §5.2 · p. 6
  • Multiple realizations improve localization and counting, but increasing from four to eight doubles inference time without significant additional performance gain.Four realizations are selected as the performance–inference-time trade-off.Source: §5.2 · pp. 6–7

6. Conclusions

CrowdDiff treats crowd density generation as denoising diffusion and uses narrow kernels, density detection, and multiple realizations to improve counting. The paper also provides training and testing pseudocode for these stages.

6. Conclusions

  • CrowdDiff treats density map generation as a denoising diffusion process and uses extremely narrow kernels to suppress noise more robustly.The framework then performs density kernel detection rather than density summation.Source: §6 · p. 7
  • Multiple stochastic density-map realizations can be combined through crowd map fusion to iteratively improve counting performance.The fusion procedure combines detected locations from multiple realizations into final locations.Source: §6 · p. 7, Figure 8 · p. 8, Table 5 · p. 8
  • The paper reports that CrowdDiff surpasses state-of-the-art performance on public datasets.The supplied conclusion also identifies separate comparisons for individual realizations, the counting branch, and crowd map fusion.Source: §6 · p. 7, Table 3 · p. 8, Table 4 · p. 8

A. Pseudocodes

  • During training, CrowdDiff samples a timestep, adds Gaussian noise to density maps, predicts that noise, and combines denoising with count-regression losses.The training pseudocode uses a hybrid loss with a weighted L1 count loss.Source: A. Pseudocodes · p. 9
  • During testing, DDIM sampling iteratively denoises initial random maps, detects head locations as contours, fuses realizations, and counts the final locations.Testing accepts a specified number of realizations and returns the count of fused locations.Source: A. Pseudocodes · p. 9

B. Experimental details

The denoising network uses a U-Net with convolutional downsampling and upsampling, self-attention, and a lightweight regression branch.

B. Experimental details

  • The denoising network is a U-Net whose spatial features are scaled by two at each downsampling and upsampling layer.Downsampling uses average pooling, while upsampling uses nearest-neighbor interpolation.Source: B. Experimental details · p. 9
  • Multi-head self-attention keeps each head's channel dimension at 64 while varying the number of heads by feature depth.The convolutional layers use 3 × 3 kernels for two-dimensional features and size-one kernels for one-dimensional features.Source: B. Experimental details · p. 9
  • The regression branch is lightweight, using linear layers, ReLU activation, global average pooling, and channel-wise concatenation.It supports the auxiliary count-estimation pathway alongside denoising.Source: B. Experimental details · p. 9

C. Evaluation metrics

Crowd counting performance is evaluated with mean absolute error and root mean squared error against ground-truth counts.

C. Evaluation metrics

  • Mean absolute error and root mean squared error are used as the performance metrics for crowd counting.The metrics compare predicted counts with ground-truth counts across test samples.Source: C. Evaluation metrics · p. 9
  • For each test sample, the ground-truth count is compared with the predicted count, and the errors are aggregated over N samples.N denotes the total number of test samples.Source: C. Evaluation metrics · p. 9

D. Datasets

The evaluation uses multiple public crowd-counting datasets spanning diverse, dense, unconstrained, and large-scale scenes.

D. Datasets

  • The study evaluates CrowdDiff on JHU-Crowd++, ShanghaiTech A, ShanghaiTech B, UCF-CC-50, and UCF-QNRF.These datasets provide varied crowd-counting benchmarks with different scales and scene characteristics.Source: D. Datasets · p. 9
  • JHU-Crowd++ includes diverse scenes, images with up to 25,791 people, and images without crowds, with 2,722 training and 1,600 test images.It also includes 500 validation images.Source: D. Datasets · p. 9
  • ShanghaiTech A and B contain 300 and 400 training images, respectively, with 182 and 316 annotated test images.Validation subsets are randomly selected from their training images.Source: D. Datasets · p. 9
  • UCF-CC-50 contains 50 extremely dense-crowd samples and uses five-fold cross-validation, while UCF-QNRF has 1,535 unconstrained crowd images.UCF-QNRF is split into 1,201 training and 334 testing images and contains approximately one million annotations.Source: D. Datasets · pp. 9–10
  • NWPU-Crowd adds a large-scale benchmark of 5,109 images split into 3,109 training, 500 validation, and 1,500 test images.It provides box-level annotations.Source: D. Datasets · p. 10

E. Additional qualitative results

The qualitative analysis finds that the training-time counting branch enriches denoising features and stabilizes predicted counts, while inference experiments examine the speed–accuracy trade-off across sampling steps.

E. Additional qualitative results

  • The counting branch produces richer decoder features and keeps predicted counts close to the ground truth across diffusion time steps.With the branch, decoder features capture crowd details from the initial time step, and predicted counts remain stable rather than deviating significantly.Source: E. Additional qualitative results · p. 10

F. Number of inference steps

  • Inference step selection balances sampling speed against MAE and MSE performance, using four realizations on UCF-QNRF.The comparison also situates CrowdDiff's inference speed against PET, STEERER, and CrowdHat.Source: F. Number of inference steps · p. 10
  • Consistency models are identified as a possible route to improving few-step sampling quality and enabling single-step inference.Source: F. Number of inference steps · p. 10

G. Training setting

The training setting reports parameters for both the denoising network and the counting decoder in a dedicated table.

G. Training setting

Related papers

Loading 2303.12790v3…