PaperlayerView exact revision on arXiv

Source-linked AI summary

TransCrowd: weakly-supervised crowd counting with transformers

Dingkang Liang, Xiwu Chen, Wei Xu, Yu Zhou, Xiang Bai

arXiv:2104.09116v3cs.CV

TL;DR

Weakly-supervised crowd counting seeks to avoid costly point-level annotations, but CNN-based image-to-count methods have limited receptive fields. TransCrowd uses a pure Transformer sequence-to-count framework and reports superior results against weakly-supervised methods with competitive performance against fully-supervised methods.

  • Problem

    Point-level annotations are expensive and redundant for evaluating counts, while weakly-supervised CNN methods use limited-receptive-field image-to-count modeling.

  • Method

    TransCrowd reformulates weakly-supervised crowd counting as sequence-to-count prediction using a pure Transformer and count-level annotations.

  • Results

    TransCrowd achieves superior counting performance against weakly-supervised methods and highly competitive performance against some fully-supervised methods on five benchmark datasets.

  • Takeaways & Limitations

    Transformer self-attention provides a promising way to capture semantic crowd information for weakly-supervised counting.

Abstract

from arXiv · show

The mainstream crowd counting methods usually utilize the convolution neural network (CNN) to regress a density map, requiring point-level annotations. However, annotating each person with a point is an expensive and laborious process. During the testing phase, the point-level annotations are not considered to evaluate the counting accuracy, which means the point-level annotations are redundant. Hence, it is desirable to develop weakly-supervised counting methods that just rely on count-level annotations, a more economical way of labeling. Current weakly-supervised counting methods adopt the CNN to regress a total count of the crowd by an image-to-count paradigm. However, having limited receptive fields for context modeling is an intrinsic limitation of these weakly-supervised CNN-based methods. These methods thus cannot achieve satisfactory performance, with limited applications in the real world. The transformer is a popular sequence-to-sequence prediction model in natural language processing (NLP), which contains a global receptive field. In this paper, we propose TransCrowd, which reformulates the weakly-supervised crowd counting problem from the perspective of sequence-to-count based on transformers. We observe that the proposed TransCrowd can effectively extract the semantic crowd information by using the self-attention mechanism of transformer. To the best of our knowledge, this is the first work to adopt a pure transformer for crowd counting research. Experiments on five benchmark datasets demonstrate that the proposed TransCrowd achieves superior performance compared with all the weakly-supervised CNN-based counting methods and gains highly competitive counting performance compared with some popular fully-supervised counting methods.

1. Introduction

TransCrowd addresses the annotation burden and limited context modeling of weakly-supervised CNN crowd counting by using a pure Transformer sequence-to-count framework. Experiments report stronger performance than weakly-supervised CNN methods and competitive performance with fully-supervised methods.

  • Motivation: Point-level annotations are costly and are not required to evaluate counting accuracy, motivating count-level weak supervision.Count-level labels provide a more economical alternative to point-level annotations.
  • Prior paradigm: Existing weakly-supervised methods directly regress the total crowd count from an image, following an image-to-count perspective.This differs from the sequence-to-count formulation proposed by TransCrowd.
  • Motivation: Transformers provide a global receptive field, addressing the limited context modeling of CNN-based weakly-supervised counting methods.The paper argues that global image perspective is suitable for predicting a total crowd count.
  • Proposed method: TransCrowd is a weakly-supervised pure Transformer framework that reformulates crowd counting as sequence-to-count prediction.It uses Transformer self-attention to extract semantic crowd information while relying only on count-level annotations.
  • Proposed method: TransCrowd-GAP uses global average pooling over Transformer output tokens, while TransCrowd-Token uses an extra learnable regression token.The authors report more reasonable attention, faster convergence, and higher counting performance for TransCrowd-GAP.
  • Results: Experiments show state-of-the-art performance against weakly-supervised methods and highly competitive performance against fully-supervised methods.The evaluation covers five benchmark datasets.

2. Related Works

Prior crowd-counting research is dominated by CNN-based localization and density-regression methods, while only a few approaches reduce annotation requirements. TransCrowd instead applies a pure Transformer to weakly-supervised sequence-to-count prediction.

  • CNN-based crowd counting: CNN-based crowd counting methods include localization-based approaches and regression-based density-map approaches.Localization methods predict human bounding boxes, whereas density-map methods integrate predicted density to obtain the total count.
  • CNN-based crowd counting: Density-map methods commonly address crowd occlusion and scale variation with multi-scale architectures and perspective-aware strategies.Examples include multi-size filters, contextual pyramids, hierarchical encoding-decoding paths, and perspective maps.
  • Attention-based methods: Attention mechanisms have been used to generate attention maps, density masks, and local or global dependencies for crowd counting.These methods remain part of the broader CNN-based counting literature.
  • Weakly-supervised counting: Only a few methods reduce annotation requirements, and some alternatives still rely on point-level annotations despite addressing limited labeled data.L2R and synthetic-data pretraining are described as examples that remain fully supervised under the paper’s definition.
  • Weakly-supervised counting: Count-level weakly-supervised methods had not achieved comparable performance to fully-supervised methods, limiting their real-world application.The paper presents TransCrowd as an alternative intended to close this performance gap.
  • Transformer-based counting: TransCrowd applies a pure Transformer with global dependencies to weakly-supervised crowd counting through a sequence-to-count formulation.The paper identifies this as the first exploration of a pure Transformer for the counting task.

3. Our Method

TransCrowd reformulates weakly-supervised crowd counting as sequence-to-count prediction: image patches become position-aware tokens, a Transformer-encoder models them, and a regression head predicts the crowd count. Two regression-head inputs are evaluated: a learnable regression token and global average pooling of visual tokens.

  • Patch Embedding: TransCrowd transforms an input image into a sequence of fixed-size flattened patches, linearly embeds them, and adds position embeddings.The resulting embedding sequence is the input to the Transformer-encoder.
  • Transformer-encoder: The Transformer-encoder uses multi-head self-attention and MLP blocks with layer normalization and residual connections.The MLP expands embeddings from D to 4D and then projects them back to D.
  • Regression Head: Training measures the difference between predicted and ground-truth crowd counts with L1 loss.Pi and Gi denote the prediction and ground-truth count for the i-th image, while M is the batch size.
  • Regression Head: A learnable regression token can aggregate overall semantic crowd information through self-attention before a two-layer MLP predicts the count.This variant is called TransCrowd-Token.
  • Regression Head: Global average pooling shrinks the visual-token sequence before a two-layer MLP regression head predicts the count.This variant is called TransCrowd-GAP, and pooled visual tokens are reported to produce richer semantic crowd patterns and better counting performance than the extra regression token.

4. Experiments

Experiments evaluate counting performance across five benchmark datasets using MAE and MSE, with datasets spanning surveillance, unconstrained, and dense crowd scenes. The supplied passages also specify dataset sizes, splits, count ranges, and core training settings.

  • Implementation Details: Training uses 384 × 384 sub-images, batch size 24, Adam optimization, ImageNet-pretrained Transformer weights, and random flipping and grayscaling augmentation.The stated learning rate is 1e-5 and weight decay is 1e-4.
  • Datasets: The experiments use NWPU-Crowd, JHU-CROWD++, UCF-QNRF, ShanghaiTech, and UCF CC 50 benchmark datasets.The datasets include large-scale, unconstrained, surveillance, and dense-crowd settings.
  • Datasets: NWPU-Crowd contains 5,109 images and 2,133,375 annotated instances, split into 3,109 training, 500 validation, and 1,500 testing images.
  • Datasets: UCF-QNRF contains 1,535 images with counts from 49 to 12,865, while ShanghaiTech contains 1,198 crowd images divided into Parts A and B.UCF-QNRF has 1,201 training and 334 testing images; ShanghaiTech provides separate training and testing splits for both parts.
  • Evaluation Metrics: Counting performance is evaluated with Mean Absolute Error (MAE) and Mean Squared Error (MSE) over testing images.Pi and Gi are the predicted and ground-truth counts for the i-th testing image.

5. Results

Across five benchmark datasets, TransCrowd delivers state-of-the-art performance among weakly-supervised methods and highly competitive performance against fully-supervised methods. It also surpasses some popular fully-supervised methods while using count-level supervision.

  • Compared with the weakly-supervised counting methods: TransCrowd achieves state-of-the-art counting performance among weakly-supervised methods across the conducted datasets.The evaluation covers five popular benchmarks and compares methods using count-level versus point-level annotations.
  • Compared with the weakly-supervised counting methods: 17.5% lower MAE and 18.8% lower MSE are reported for TransCrowd-GAP versus MATT on ShanghaiTech part A.
  • Compared with the weakly-supervised counting methods: 20.5% lower MAE and 8.0% lower MSE are reported for TransCrowd-GAP versus MATT on ShanghaiTech part B.
  • Compared with the weakly-supervised counting methods: TransCrowd-Token also significantly improves MAE and MSE over MATT and, with performance close to fully-supervised methods.MATT still uses a small number of images containing point-level annotations for training.
  • Compared with the fully-supervised counting methods: On JHU-Crowd++ testing, TransCrowd-GAP improves 11.0 MAE and 13.6 MSE over CSRNet, while improving 0.1 MAE and 4.3 MSE over BL.The paper also reports significant improvements over some popular fully-supervised methods on UCF-QNRF, ShanghaiTech, and NWPU-Crowd.
  • Compared with the fully-supervised counting methods: The results indicate that point-level annotations are not entirely necessary for the counting task.

6. Analysis

The analysis compares TransCrowd variants, convergence, runtime, pre-training, transferability, and attention behavior. TransCrowd-GAP generally shows faster convergence, higher counting performance, and more reasonable attention than TransCrowd-Token.

  • Attention analysis: TransCrowd-GAP generates more reasonable attention maps than TransCrowd-Token, which may attend more to background regions.Both variants successfully focus on crowd regions.
  • Convergence analysis: TransCrowd-GAP achieves better performance than CSRNet with 1.9 × fewer training epochs.Both TransCrowd variants have smooth, fast-converging curves, whereas CSRNet’s curve oscillates.
  • Regression-head comparison: Global average pooled visual tokens converge faster and achieve better count accuracy than an extra regression token.This comparison supports TransCrowd-GAP over TransCrowd-Token in the regression head design.
  • Runtime comparison: Despite containing more parameters, TransCrowd-Token and TransCrowd-GAP achieve outstanding runtime compared with fully supervised methods.The analysis attributes this to fully supervised methods maintaining high-resolution features for density-map generation.
  • Pre-training comparison: With GCC pre-training, TransCrowd-GAP achieves better counting performance than CSRNet and can outperform several recent fully supervised methods.Without pre-training, the CNN-based method outperforms the Transformer-based method; GCC is synthetic and uses count-level annotations here.
  • Transferability: TransCrowd-GAP achieves highly competitive cross-dataset performance without further fine-tuning, indicating remarkable transferability.The cross-dataset evaluation uses UCF-QNRF and ShanghaiTech Parts A and B as source and target datasets.

7. Conclusion

The paper introduces TransCrowd, a Transformer-encoder framework that reformulates weakly-supervised crowd counting as sequence-to-count prediction. Across five challenging datasets, it outperforms state-of-the-art weakly-supervised methods and remains competitive with popular fully supervised methods.

  • Conclusion: TransCrowd presents a sequence-to-count perspective for weakly-supervised crowd counting using a Transformer-encoder.The framework is named TransCrowd.
  • Conclusion: The attention mechanism is promising for capturing semantic crowd information in the counting task.The paper identifies this as a property of the proposed Transformer-based approach.
  • Conclusion: Experiments on five challenging datasets show superior performance against state-of-the-art weakly-supervised methods and competitive performance against popular fully supervised methods.The comparison is reported as the paper’s overall experimental conclusion.
Loading 2104.09116v3…