PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixel-wise Crowd Understanding via Synthetic Data

Qi Wang, Junyu Gao, Wei Lin, Yuan Yuan

arXiv:2007.16032v2cs.CV

TL;DR

Pixel-wise crowd understanding needs extensive labels, but crowd-scene annotation is costly. The paper builds the synthetic GCC dataset and uses it for supervised pre-training and domain adaptation; experiments report improved real-world counting and domain-adaptation performance without real-data labels.

  • Problem

    Pixel-wise crowd tasks need substantial labeled data, yet annotation is costly and crowd datasets remain limited.

    Source: §2.3 · p. 4

  • Method

    The paper constructs automatically labeled synthetic GCC scenes and uses them for real-data fine-tuning or for translation-based domain adaptation with unlabeled real images.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Counting performance improves by an average of 12%, and the domain-adaptation method outperforms two baselines addressing the same problem.

    Source: §1 · p. 3, §7 · p. 17

  • Takeaways & Limitations

    The synthetic-data approaches improve crowd-model performance in real scenes, while domain adaptation avoids the burden of manually labeling real data.

    Source: §1 · p. 3, §7 · p. 17

    In extremely congested scenes, translated images lose texture, sharpness, and edge detail, and predicted maps can diverge substantially from ground truth.

    Source: Crowd Counting via Domain Adaptation · p. 16

Abstract

from arXiv · show

Crowd analysis via computer vision techniques is an important topic in the field of video surveillance, which has wide-spread applications including crowd monitoring, public safety, space design and so on. Pixel-wise crowd understanding is the most fundamental task in crowd analysis because of its finer results for video sequences or still images than other analysis tasks. Unfortunately, pixel-level understanding needs a large amount of labeled training data. Annotating them is an expensive work, which causes that current crowd datasets are small. As a result, most algorithms suffer from over-fitting to varying degrees. In this paper, take crowd counting and segmentation as examples from the pixel-wise crowd understanding, we attempt to remedy these problems from two aspects, namely data and methodology. Firstly, we develop a free data collector and labeler to generate synthetic and labeled crowd scenes in a computer game, Grand Theft Auto V. Then we use it to construct a large-scale, diverse synthetic crowd dataset, which is named as "GCC Dataset". Secondly, we propose two simple methods to improve the performance of crowd understanding via exploiting the synthetic data. To be specific, 1) supervised crowd understanding: pre-train a crowd analysis model on the synthetic data, then fine-tune it using the real data and labels, which makes the model perform better on the real world; 2) crowd understanding via domain adaptation: translate the synthetic data to photo-realistic images, then train the model on translated data and labels. As a result, the trained model works well in real crowd scenes.

1 Introduction

Pixel-wise crowd understanding combines counting and segmentation, but scarce, costly labels and limited scene diversity constrain training data. The paper introduces the synthetic GCC dataset and two ways to use it to improve crowd models on real scenes.

1 Introduction

  • Crowd counting estimates regional density and people counts, while segmentation predicts crowd regions; together they form pixel-wise crowd understanding.Pixel-level density estimation and localization also provide attention information for higher-level analysis such as person-group detection.Source: §1 · pp. 1–2
  • Existing crowd datasets are small because pixel-wise annotation is costly; some congested-scene head labels are inaccurate, and common conditions such as night are underrepresented.Labeling a scene with 1,000 people takes more than 40 minutes.Source: §1 · p. 2
  • The supervised strategy pre-trains a fully convolutional crowd model on GCC, then fine-tunes it on labeled real-world data.The authors contrast this initialization with random initialization and ImageNet pre-training.Source: §1 · p. 2, Figure 1 · p. 2
  • For domain adaptation, CycleGAN translates GCC images into photo-realistic scenes, then adversarial training uses translated labels and unlabeled real images.The translation adds an SSIM loss to retain crowd-region texture and local patterns.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The proposed SFCN achieves state-of-the-art results, and domain adaptation outperforms two baselines without using real-data labels.The paper reports that supervision outperforms state of the art on four real datasets.Source: §1 · p. 3
  • GCC is generated and labeled automatically in GTA V; it contains 15,212 images and 7,625,843 people, with head-dot and crowd-mask labels.The dataset is described as free to collect and annotate, larger, more diverse, and more accurately labeled than existing real-world datasets.Source: §1 · pp. 2–3
  • This paper extends earlier crowd-counting work with faster scene generation, segmentation masks, a segmentation network, joint adversarial training, and further real-world experiments.The optimized generation process reduces simulation time by two-thirds.Source: §1 · p. 3

2 Related Works

Related work covers pixel-wise crowd models, real crowd datasets, and synthetic-data platforms. It traces methodological approaches and dataset development while motivating synthetic data as a response to expensive annotation.

2.1 Pixel-wise Crowd Understanding

  • Pixel-wise crowd models commonly use FCNs for density maps or crowd regions, with multi-scale features, multi-task learning, or contextual information addressing modeling challenges.Other approaches include patch-level regression, multi-column networks, multi-task learning, and methods for encoding large-range context.Source: §2.1 · p. 3
  • Prior work addresses scarce training data with web-data self-supervision, negative-correlation learning, and a dense counting autoencoder trained mostly without labels.The autoencoder trains 99.9% of its parameters without labeled data.Source: §2.1 · pp. 3–4

2.2 Crowd Datasets

  • Crowd datasets range from sparse single-scene collections to congested and cross-scene datasets, but the reviewed datasets retain drawbacks noted by the authors.The cited table compares eight real-world datasets with GCC by volume, resolution, and label forms.Source: §2.2 · p. 4, Table 1 · p. 4

2.3 Synthetic Dataset

  • Pixel-wise annotation is labor-intensive, motivating synthetic datasets and low-cost annotation methods across virtual worlds and game-engine-based applications.Annotating a crowd scene with 1,000 people takes more than 40 minutes; prior examples include GTA V street scenes and synthetic segmentation data.Source: §2.3 · p. 4
  • Synthetic-data tools include semi-realistic virtual-world platforms and simulators such as CARLA, UnrealCV, and AirSim, which offer configurable environments or visual data outputs.AirSim supports RGB images, depth information, and pixel-level segmentation masks.Source: §2.3 · p. 5

3 GTA5 Crowd Counting (GCC) Dataset

The authors build a synthetic crowd dataset in GTA V to address the scarcity of labeled crowd scenes, then describe its diversity and evaluation splits. Their collector constructs congested scenes and automatically labels head locations.

3 GTA5 Crowd Counting (GCC) Dataset

  • Earlier synthetic GTA V datasets lacked congested scenes, so the authors built a collector and labeler to construct crowds and annotate head locations automatically.The process controls pedestrians and virtual-world attributes, then analyzes rendering-stencil data for labels.Source: §3 · pp. 5–6

3.1 Data Collection

  • Scene synthesis combines pedestrian regions into a crowd scene, saves images, masks, and head coordinates, then removes labels for occluded heads.The process addresses GTA V’s limit of 256 people by placing people in several non-overlapping regions.Source: §3.1 · p. 6

3.2 Properties of GCC

  • GCC contains 15,212 images at 1080×1920 resolution and more than 7,000,000 people across 400 varied scenes.The dataset includes locations such as offices, stores, malls, sidewalks, plazas, and stadiums.Source: §3.2 · p. 6
  • Randomly assigning crowd counts across each scene’s capacity range gives GCC a wider crowd-count range than other real datasets.The authors note that existing datasets focus on sparse or congested crowds, while large scenes in the wild may also have few people.Source: §3.2 · p. 6
  • GCC varies scene capacity, timestamp, and weather; its generation favors daytime and fine-weather scenes.The dataset includes random capture times and weather conditions, including clear, cloudy, rainy, foggy, thunder, overcast, and extra-sunny conditions.Source: §3.2 · pp. 6–7, Figure 3 · p. 7
  • The evaluation splits are random, cross-camera, and cross-location; the latter two are intended to assess model generalization.Random splitting uses 75% of images for training and 25% for testing; cross-location splitting uses 75% of locations for training and 25% for testing.Source: §3.2 · p. 7

4 Supervised Crowd Understanding

The supervised approach uses a spatial FCN for crowd counting and segmentation, with pretraining on GCC followed by fine-tuning on real data. The authors propose this initialization to address overfitting from scarce real labels.

4 Supervised Crowd Understanding

  • The proposed Spatial FCN targets both crowd counting and segmentation.The architecture illustration is provided in Fig. 6.Source: §4 · p. 7, Figure 6 · p. 8

4.1 Spatial FCN for Crowd Understanding

  • SFCN adds a spatial encoder to a backbone, then predicts a density map at one-eighth input size and a full-resolution crowd mask.Counting uses pixel-wise mean squared error; segmentation uses 2-D cross-entropy.Source: §4.1 · pp. 7–8

4.2 Pretraining & Fine-tuning

  • Pre-GCC pretrains the designed model on GCC before fine-tuning on real data, initializing the full model rather than only its backbone.The authors state that this can alleviate overfitting more effectively than ImageNet pretraining, which initializes only the backbone.Source: §4.2 · p. 8

5 Crowd Understanding via Domain Adaptation

The domain-adaptation approach uses labeled synthetic images and unlabeled real images to train a counter for real-world density maps. It combines image translation, domain-invariant feature learning, and data and output constraints.

5.1 SSIM Embedding CycleGAN

  • The method translates synthetic images toward photo-realistic images, trains an SFCN counter to learn domain-invariant features, and applies it directly to real data.Source: §5.1 · p. 8, §5 · p. 8

5.1.1 CycleGAN

  • CycleGAN uses generators for synthetic-to-real and real-to-synthetic mappings, cycle consistency, and adversarial discriminators to translate between domains.Source: §5.1.1 · p. 8

5.1.2 SSIM Embedding CycleGAN

  • SE CycleGAN adds an SSIM loss to preserve local structured features that ordinary cycle consistency may lose during translation.Source: §5.1.2 · pp. 8–9

5.1.3 Feature-level Adversarial Learning

  • A domain classifier adversarially distinguishes synthetic from real encoder features, encouraging the encoders to learn domain-invariant representations.Source: §5.1.3 · p. 9

5.2 Joint Training

  • Joint training combines counting loss on translated synthetic data, translation loss, and inverse adversarial loss on real-domain features, balanced by weights α, β, and λ.Source: §5.2 · p. 9

5.3 Scene/Density Regularization

  • Scene regularization coarsely filters translated synthetic scenes to better match each target dataset, since irrelevant scenes can reduce adaptation performance.Source: §5.3 · pp. 9–10
  • Density regularization sets predicted pixels above MAXS—the maximum density in synthetic data—to zero, addressing exceptionally large background estimates.Source: §5.3 · p. 10

6 Experiments

The experiments evaluate crowd counting and segmentation, including training within GCC and pre-training on GCC followed by fine-tuning on real datasets. The section specifies the metrics and implementation choices used for these evaluations.

6.1 Metrics

  • Counting is evaluated with MAE and MSE, with PSNR and SSIM also used to assess density-map quality.Source: §6.1 · p. 10
  • Segmentation performance is evaluated with IoU for both crowd and background, calculated from true-positive, false-positive, and false-negative counts.Source: §6.1 · p. 10

6.2 Results of Supervised Crowd Understanding

  • Experiments cover training and testing within GCC and pre-training on GCC followed by fine-tuning on real datasets; models are trained with Adam, using a 10% validation split.Source: §6.2 · p. 10, §6.2.1 · p. 10

6.2.1 Implementation Details

Performance of Overall Evaluation

On GCC, SFCN outperforms the VGG-16-based comparison models across seven counting and segmentation metrics, while multi-task learning also beats single-task learning. Across real-dataset fine-tuning comparisons, pre-training on counting data reduces average error, with Pre-GU performing best among the listed strategies.

Performance of Overall Evaluation

6.2.3 Comparison of Different Pre-trained Models

  • Pre-GCC, Pre-UR, and Pre-GU reduce average error by 15%, 14%, and 20%, respectively, and Pre-GU performs better than the other two strategies.Source: §6.2.3 · p. 12, Table 5 · p. 12
  • Average improvement is approximately 31% for scratch-trained MCNN versus approximately 12% for ImageNet-pretrained CSRNet, SFCN, and SFCN†.Source: §6.2.3 · p. 12

Comparison on the Parameter Level

The parameter-level comparison examines how pre-training changes SFCN’s layer-wise weight and bias distributions, alongside visual density-map estimates during training.

Comparison on the Parameter Level

Visual Comparison of Different Pre-trained Models

Pre-GCC helps SFCN converge faster and produce better density estimates than Pre-ImgNt, especially early in training.

Visual Comparison of Different Pre-trained Models

  • Pre-GCC converges faster than Pre-ImgNt and achieves better training results, with its loss curve lower during training.Before Epoch 10, Pre-GCC produces acceptable density maps while Pre-ImgNt outputs only coarse distributions; by Epoch 200, Pre-ImgNt improves but remains worse.Source: Visual Comparison of Different… · p. 13

Generalization Ability of Different Pre-trained Models

Pre-GCC improves cross-dataset counting generalization and performs strongly against state-of-the-art methods. Fine-tuning benefits from more pre-training data and from data whose density and luminance better match the real datasets; the domain-adaptation experiments specify their target datasets and image-selection setup.

Generalization Ability of Different Pre-trained Models

6.2.4 The Effect of Pre-GCC on the Fine-tuning Results

  • As GCC pre-training data increases from 20% to 100%, fine-tuning SFCN’s MAE and MSE decrease on UCF-QNRF and SHT B.The experiments compare different GCC data volumes; Fig. 11 identifies the fine-tuning results on those datasets.Source: §6.2.4 · p. 14, Figure 11 · p. 14
  • For UCF-QNRF and SHT B, pre-training on denser GCC scene levels performs better; SHT B’s lowest reported errors are MAE/MSE 9.1/14.8 with {L5, L6}.The authors conclude that fine-tuning performs better when pre-training density distributions are closer to those of the real data.Source: §6.2.4 · p. 14, Table 7 · p. 14
  • Low-luminance GCC pre-training yields larger final estimation errors than high-luminance pre-training on UCF-QNRF and SHT B, whose data rarely include low-luminance scenes.The authors attribute this difference to high-luminance GCC being closer to the real data.Source: §6.2.4 · p. 14, Table 8 · p. 14
  • The fine-tuning analysis concludes that more pre-training data should be as similar as possible to the real data, while selection of proper pre-training data remains an open question.Source: §6.2.4 · p. 14

6.2.5 Comparison with the SOTA Methods

  • SFCN† with Pre-GCC refreshes six records across nine metrics on five datasets and achieves the best MAE on UCF-QNRF (102.0), SHT A (64.8), SHT B (7.6), and UCF CC 50 (214.2).Table 9 compares performance on UCF-QNRF, SHT A/B, UCF CC 50, and WorldExpo’10.Source: §6.2.5 · p. 14, Table 9 · p. 15

6.3.1 Implementation Details

  • Scene Regularization selects candidate images by congestion ratio, with the stated filter conditions listed for the adaptation setup.The passage illustrates the restriction by excluding an SHT A candidate whose crowd count falls outside the selected ratio range.Source: §6.3.1 · p. 15, Table 10 · p. 15

6.3.2 Adaptation Performance on Real-world Datasets

  • The domain-adaptation experiments cover five real-world counting datasets and CityScapes segmentation, using Scene/Density Regularization for the reported models.The counting models include NoAdpt, CycleGAN, SE CycleGAN, and jointly trained SE CycleGAN with SFCN.Source: §6.3.2 · p. 15

Crowd Counting via Domain Adaptation

Across five real-world counting datasets, domain adaptation outperforms no adaptation, and SE CycleGAN generally outperforms CycleGAN. Joint training improves most metrics, while extremely congested scenes remain a challenge.

Crowd Counting via Domain Adaptation

  • On SHT B, SE CycleGAN achieves MAE/MSE of 16.4/25.8, which the authors attribute to consistent sensor capture and high image clarity.The authors report that this result outperforms some early supervised methods (Zhang et al., 2016b; Sindagi and Patel, 2017a; Sam et al., 2017; Sindagi and Patel, 2017b).Source: Crowd Counting via Domain Adaptation · pp. 15–16
  • Adaptation performs far better than NoAdpt across the reported metrics, SE CycleGAN generally outperforms CycleGAN, and joint training improves most metrics.The comparison covers four metrics, except for MAE alone on WorldExpo’10; PSNR decreases on Shanghai Tech A and UCF CC 50 after adding SSIM loss.Source: Crowd Counting via Domain Adaptation · p. 15, Table 11 · p. 16
  • In extremely congested scenes, jointly trained predictions remain far from ground truth; the authors suggest lost texture, sharpness, and edge details in translated images as the main reason.Fig. 12 shows visualized results on SHT A and B.Source: Crowd Counting via Domain Adaptation · p. 16, Figure 12 · p. 16

Crowd Segmentation via Domain Adaptation

Domain adaptation improves crowd segmentation on CityScapes, while scene filtering and SSIM embedding address separate aspects of adapting synthetic crowd data to real scenes.

Crowd Segmentation via Domain Adaptation

6.3.3 The Effectiveness of SDR

  • Scene/Density Regularization improves adaptation performance by filtering synthetic scenes that do not exist in the real datasets.The comparison evaluates NoAdpt, CycleGAN, and SE CycleGAN on SHT A with and without SDR.Source: Table 13 · p. 17, §6.3.3 · p. 17

6.3.4 Analysis of SSIM Embedding

  • SE CycleGAN preserves structural similarity during translation and produces more consistent images, retaining more texture and contrast and fewer abnormal color values than CycleGAN.The paper’s visual comparison marks image regions with colored boxes to illustrate these differences.Source: §6.3.4 · p. 17, Figure 13 · p. 17

7 Conclusion and Outlook

The paper reports two uses of synthetic data for real-world crowd understanding: GCC pre-training improves counting, while domain adaptation performs better than no adaptation. The outlook focuses on reusable data-generation tools and domain-adaptive methods as paths toward practical crowd understanding.

7 Conclusion and Outlook

  • Counting performance improves by an average of 12% with the proposed pre-training scheme compared with traditional ImageNet pre-training.The domain-adaptation model works better than the traditional no-adaptation method; in ShanghaiTech Part B, WorldExpo’10, UCSD, and Mall, its estimates are very close to supervised learning.Source: §7 · p. 17
  • The authors identify unsupervised or few-shot domain adaptation as a more practical research area than supervised learning because it can reduce real-scene data collection and annotation costs.Source: §7 · p. 18
  • The open-sourced tools can be adapted to generate synthetic image or video data for tasks including counting, localization, tracking, instance segmentation, and trajectory prediction.Source: §7 · p. 18
  • Future work will focus on data generation and domain-adaptive tasks to promote practical crowd understanding in the real world.Source: §7 · p. 18
Loading 2007.16032v2…