PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Synthetic Data for Crowd Counting in the Wild

Qi Wang, Junyu Gao, Wei Lin, Yuan Yuan

arXiv:1903.03303v1cs.CV

TL;DR

Crowd counting in the wild is hindered by scarce, imperfectly labeled data and variation in scene conditions and crowd sizes. The paper constructs a large synthetic dataset with automatic annotations and uses it for supervised pretraining plus domain adaptation; supervised learning reaches state-of-the-art performance, while domain adaptation obtains acceptable results and outperforms its baselines.

  • Problem

    Crowd counting in the wild requires large, accurately labeled, diverse datasets, but existing data are scarce and current datasets have limited coverage of crowd conditions.

  • Method

    The paper develops an automatic GTA5 data collector and labeler, constructs the GCC synthetic dataset, and exploits it through supervised pretraining and SE Cycle GAN-based domain adaptation.

  • Results

    Supervised learning achieves state-of-the-art performance on real crowd-counting datasets, while the domain-adaptation method obtains acceptable results and outperforms two baselines.

  • Takeaways & Limitations

    Synthetic data can support crowd counting in the wild through both real-data fine-tuning and domain adaptation without labels for real data.

  • Takeaways & Limitations

    Original cycle consistency can lose local patterns and texture features during synthetic-to-real translation, causing detailed information loss and distortion.

Abstract

from arXiv · show

Recently, counting the number of people for crowd scenes is a hot topic because of its widespread applications (e.g. video surveillance, public security). It is a difficult task in the wild: changeable environment, large-range number of people cause the current methods can not work well. In addition, due to the scarce data, many methods suffer from over-fitting to a different extent. To remedy the above two problems, firstly, we develop a data collector and labeler, which can generate the synthetic crowd scenes and simultaneously annotate them without any manpower. Based on it, we build a large-scale, diverse synthetic dataset. Secondly, we propose two schemes that exploit the synthetic data to boost the performance of crowd counting in the wild: 1) pretrain a crowd counter on the synthetic data, then finetune it using the real data, which significantly prompts the model's performance on real data; 2) propose a crowd counting method via domain adaptation, which can free humans from heavy data annotations. Extensive experiments show that the first method achieves the state-of-the-art performance on four real datasets, and the second outperforms our baselines. The dataset and source code are available at https://gjy3035.github.io/GCC-CL/.

1. Introduction

Crowd counting in the wild is limited by insufficiently diverse and accurately labeled data, especially under extreme conditions. The paper addresses this with the synthetic GCC dataset and two methods that exploit it for real-world counting.

  • Crowd counting supports video surveillance, public-area planning, and traffic-flow monitoring by predicting density maps and estimating people counts.
  • Existing datasets inadequately cover changeable weather, variant illumination, and wide-ranging crowd sizes, limiting performance on unseen wild cases.
  • Rare labeled data contributes to overfitting and performance degradation when crowd-counting algorithms transfer to wild or other scenes.
  • The GCC dataset uses an automatic collector and labeler to generate large-scale, diverse synthetic crowd scenes with accurate annotations without manual labor.
  • The paper pretrains SFCN on synthetic data before real-data fine-tuning, reducing estimation errors and achieving state-of-the-art results.
  • The domain-adaptation method uses SE Cycle GAN to translate synthetic scenes into photo-realistic images, enabling training without real-data labels and outperforming two baselines.

2. Related Works

Related work spans crowd-counting architectures, benchmark datasets, and synthetic data intended to reduce annotation costs. The paper situates GCC among these approaches and datasets.

  • Crowd Counting Methods: CNN-based crowd-counting methods improve performance through multi-task learning, multi-stream or multi-scale features, and large-range contextual information.
  • Crowd Counting Methods: Self-supervised learning and deep negative correlation learning have been explored to address scarce crowd-counting data.
  • Crowd Counting Datasets: Existing datasets cover settings including sparse pedestrian walkways, shopping malls, highly congested scenes, surveillance footage, and extremely congested real-world images.
  • Synthetic Dataset: Synthetic datasets are introduced to reduce the time and labor required for ground-truth annotation, including datasets based on GTA5 scenes.
  • Synthetic Dataset: Figure 2 presents GCC examples organized by scene capacity, timestamp, and weather conditions.
  • Synthetic Dataset: Figure 3 demonstrates combining images to construct congested crowd scenes.

3. GTA5 Crowd Counting (GCC) Dataset

The GCC dataset uses GTA5 to generate and automatically annotate synthetic crowd scenes, targeting diverse crowd counts and environmental conditions for crowd counting in the wild.

  • Data collection: GTA5’s virtual world supports synthetic crowd-scene collection with automatic head-location annotation from rendering stencil data.The collector constructs congested scenes, captures stable images, and the labeler annotates person heads without manual labor.
  • Dataset scale: 15,212 images and 7,625,843 persons make GCC larger than existing crowd-counting datasets.Each image has resolution 1080 × 1920.
  • Diverse scenes: GCC includes 400 scenes spanning indoor and outdoor locations, with scene levels controlling the range of crowd sizes.The dataset includes locations such as convenience stores, pubs, malls, streets, plazas, and stadiums.
  • Diverse scenes: Randomized crowd sizes give GCC a larger crowd-count range than real datasets that focus mainly on sparse or congested scenes.The number of people is sampled within each scene level’s range.
  • Diverse environments: GCC varies capture time and weather using seven weather types to approximate changing conditions in the wild.Images are generated under randomized times and weather conditions, including clear, clouds, rain, foggy, thunder, overcast, and extra sunny.

4. Supervised Crowd Counting

The supervised approach uses SFCN to regress density maps and pretrains it on GCC before fine-tuning on real datasets, addressing scarce labeled data and transfer performance.

  • Model: SFCN directly regresses density maps while encoding global context through a spatial fully convolutional architecture.It uses VGG-16 or ResNet-101 as a backbone and outputs a density map at one-eighth the input size.
  • Experimental design: Pretraining on GCC followed by real-data fine-tuning is evaluated alongside training within GCC under multiple split strategies.The GCC experiments use random, cross-camera, and cross-location splits.
  • Pretraining: The GCC-pretrained model is fine-tuned with all parameters trained, rather than relying on random initialization or ImageNet classification pretraining.The comparison includes MCNN, CSR, and SFCN on UCF-QNRF and SHT B.
  • Results: SFCN† fine-tuned from GCC-pretrained weights achieves state-of-the-art results on four of five real datasets.The detailed comparisons are reported in Table 4.

5. Crowd Counting via Domain Adaptation

The proposed domain-adaptation pipeline transfers crowd-counting knowledge from labeled synthetic images to unlabeled real images. It combines SE Cycle GAN translation, SFCN counting, and density/scene regularization to reduce domain gaps and preserve crowd details.

  • Overview: The method trains SE Cycle GAN to translate synthetic images into photo-realistic images, then trains SFCN on translated data and tests directly on real data without manual labels.The synthetic domain provides images and count labels, whereas the real domain provides images only.
  • SSIM Embedding Cycle GAN: SSIM Embedding adds an SSIM penalty to Cycle GAN cycle consistency, helping preserve local patterns and texture information during translation.The original cycle consistency can lose details and distort translated images, especially in congested regions.
  • SSIM Embedding Cycle GAN: The adaptation objective combines adversarial loss, cycle-consistent loss, and SSIM Embedding cycle-consistent loss.The cycle-consistent and SSIM Embedding terms are weighted by λ and µ, with µ set to 1 during training.
  • Density/Scene Regularization: Density/scene regularization bounds predicted density values by the maximum synthetic density and filters synthetic scenes that are unsuitable for specific real datasets.The scene filter is described as a coarse data filter rather than an elaborate selection procedure.
  • Performance on Real-world Datasets: Adaptation performs far better than no adaptation, while SE Cycle GAN generally outperforms original Cycle GAN across five real-world datasets.Only two PSNR reductions are reported for SE Cycle GAN, on Shanghai Tech A and UCF CC 50.
  • Visualization: Visualizations show Cycle GAN improves map quality over no adaptation, but extremely congested scenes remain difficult when translated details are lost.SE Cycle GAN retains more local texture and structural information than Cycle GAN, although both adaptation methods have side effects.

6. Conclusion

The paper addresses crowd counting in the wild by constructing a large-scale synthetic dataset and exploiting it through supervised learning and domain adaptation. The supervised method achieves state-of-the-art performance, while the domain-adaptation method obtains acceptable results.

  • Conclusion: The paper develops an automatic data collector and labeler, constructs a large-scale synthetic dataset, and uses synthetic data for supervised learning and domain adaptation.These methods are presented as ways to improve crowd-counting performance in the wild.
  • Conclusion: The supervised method achieves state-of-the-art performance, while the domain-adaptation method obtains acceptable results.The authors identify more effective domain-invariant features as future work.

Supplementary

The supplementary material provides additional information about the dataset, supervised method, and domain-adaptation method, corresponding to Sections 3, 4, and 5.

  • Supplementary: The supplementary material covers dataset, supervised, and domain-adaptation information corresponding to Sections 3, 4, and 5.It is organized around three perspectives of the work.

7. GCC Dataset

The GCC dataset is illustrated through representative synthetic scenes and GTA5-world locations, with reproducible camera and region-of-interest metadata. Each crowd image also includes contextual annotations beyond head coordinates.

  • Dataset Examples: Figure 10 presents representative synthetic crowd scenes from the GCC dataset.The supplementary material uses these examples to provide a deeper understanding of GCC.
  • Reproducibility: GCC scenes can be reproduced from camera parameters and polygonal regions of interest used for placing person models.Provided camera metadata includes position, height, pitch/yaw angle, and field of view; each scene also receives a nine-level capacity label.
  • Annotations: Each crowd image includes head coordinates, capture time in 24-hour format, weather condition, and a binary crowd-segmentation map.These annotations supplement the person-location information.
  • Locations: Figure 11 shows 100 selected GTA5-world locations, mainly concentrated in urban areas.The locations demonstrate the spatial coverage used for data collection.

8. Supervised Crowd Counting

This section specifies the FCN, SFCN, and SFCN† configurations and evaluates SFCN on GCC using MAE and MSE. Performance varies with split difficulty, scene size, time of day, and weather.

  • Architecture: Table 7 compares the network architectures of FCN, SFCN, and SFCN†.The notation k(3,3)-c256-s1-d2 denotes a 3 × 3 kernel, 256 output channels, stride 1, and dilation rate 2.
  • Evaluation: SFCN results on GCC are reported using standard MAE and MSE, with Average denoting the average value for each class.
  • Split difficulty: MAE and MSE increase from random to cross-camera to cross-location splitting, indicating progressively harder evaluation tasks.
  • Scene conditions: Small scenes are easier to count than large scenes because large scenes cover a much wider range of people counts.
  • Scene conditions: Daytime scenes are easier to count than night scenes, while clear, cloudy, overcast, and extra-sunny scenes are easier than rainy, foggy, and thunder scenes.

9. Crowd Counting via Domain Adaptation

The domain-adaptation section filters synthetic images by scene conditions and compares original Cycle GAN with SE Cycle GAN when translating GCC scenes to real-world domains. SE Cycle GAN produces more consistent, textured, color-stable, and contrast-preserving images in the reported visual comparisons.

  • Scene Regularization: Scene Regularization selects images for adaptation to five real datasets using a coarse data filter.
  • Scene Regularization: The ratio range restricts selected images by congestion, requiring their people counts to fall within a specified fraction of the target level range.For SHT A, 800 people is excluded from level 0∼4000 under a ratio range of 0.5∼1.
  • Scene Regularization: The adaptation table categorizes people-count levels from 0∼10 through 0∼4k and weather as clear, clouds, rain, foggy, thunder, overcast, or extra sunny.
  • Visual comparison: Figures 12–14 compare translations from the synthetic source domain to five real-world target datasets, with original Cycle GAN above and SE Cycle GAN below.
  • Visual comparison: SE Cycle GAN produces more consistent images, preserves more texture and original-image contrast, and avoids abnormal color values relative to Cycle GAN.
  • Visual comparison: Overall, the proposed SE Cycle GAN generates higher-quality crowd scenes than the original Cycle GAN in the visualization results.
Loading 1903.03303v1…