PaperlayerView exact revision on arXiv

Source-linked AI summary

CCTrans: Simplifying and Improving Crowd Counting with Transformer

Ye Tian, Xiangxiang Chu, Hongpeng Wang

arXiv:2109.14483v1cs.CV

TL;DR

Crowd counting must handle scale and density variation, while CNNs have limited receptive fields for global context. CCTrans uses a pyramid transformer, feature aggregation, and multi-scale regression with tailored losses; it reports state-of-the-art results in both supervision settings and ranks No.1 on NWPU-Crowd.

Abstract

from arXiv · show

Most recent methods used for crowd counting are based on the convolutional neural network (CNN), which has a strong ability to extract local features. But CNN inherently fails in modeling the global context due to the limited receptive fields. However, the transformer can model the global context easily. In this paper, we propose a simple approach called CCTrans to simplify the design pipeline. Specifically, we utilize a pyramid vision transformer backbone to capture the global crowd information, a pyramid feature aggregation (PFA) model to combine low-level and high-level features, an efficient regression head with multi-scale dilated convolution (MDC) to predict density maps. Besides, we tailor the loss functions for our pipeline. Without bells and whistles, extensive experiments demonstrate that our method achieves new state-of-the-art results on several benchmarks both in weakly and fully-supervised crowd counting. Moreover, we currently rank No.1 on the leaderboard of NWPU-Crowd. Our code will be made available.

Introduction

CCTrans asks whether transformers can simplify crowd-counting pipelines while addressing scale and density variation and the limited receptive fields of CNNs. It combines transformer features, feature aggregation, and multi-scale regression, with state-of-the-art results in both supervision settings.

Introduction

  • Crowd-counting scenes vary in scale and density, while CNNs’ limited receptive fields constrain global-context modeling.Multi-column models can be inefficient, and attention-based pipelines can require sensitive hyperparameter tuning.Source: Introduction · p. 1
  • The paper asks whether transformers can simplify crowd-counting pipelines, motivated by their ability to capture long-context dependencies and global receptive fields.The cited transformer-based segmentation models report better performance than CNN baselines.Source: Introduction · p. 1
  • CCTrans combines a transformer for global-context features with feature aggregation, multi-scale regression, and tailored losses for weakly and fully-supervised counting.The contribution list names smooth weighted loss for weak supervision and smooth L1 for full supervision.Source: Introduction · p. 1, Figure 1 · p. 2
  • CCTrans achieves new state-of-the-art results on UCF QNRF and NWPU-Crowd under both weakly and fully-supervised settings.The paper also reports that CCTrans ranks No.1 on the NWPU-Crowd leaderboard.Source: Introduction · p. 2

Proposed Method

The method converts input images into patch sequences, extracts global features with a pyramid transformer, and aggregates stage features as 2D maps. A multi-scale regression head predicts density maps used for supervision.

Proposed Method

  • Input images are split into patches and flattened into sequences for global feature extraction by a pyramid transformer.Stage sequences are reshaped into 2D maps, upsampled to a shared resolution, and combined by elementwise addition.Source: Proposed Method · p. 2
  • A regression head with multi-scale receptive fields predicts the density map from the aggregated features.For weak supervision, the predicted map’s pixel-value sum is used in the loss; for full supervision, the density map is used.Source: Proposed Method · p. 2

2D Image to 1D Sequence

CCTrans converts image patches to transformer features, fuses shallow detail with deep semantics, and regresses density maps through a multi-scale head. The section also adapts loss design to fully and weakly supervised counting.

2D Image to 1D Sequence

Transformer Backbone

  • Twins alternates local and global attention to capture short- and long-range relations, while its separable attention reduces computation.Source: Transformer Backbone · p. 2

Pyramid Feature Aggregation

Regression Head with Multi-scale Receptive Fields

Loss Function Design

  • Fully supervised loss adds L2 regularization on smoothed maps in place of TV's rough head-annotation regularization, while weak supervision uses smooth L1 for count outliers; both designs report gains.Source: Loss Function Design · pp. 3–4

Experiments

Experiments evaluate CCTrans on five crowd-counting benchmarks and report state-of-the-art comparisons, then assess its components and loss choices through ablations. The ablations highlight contributions from feature aggregation, multi-scale convolution, and tailored losses.

Implementation Details.

  • Evaluation covers five benchmarks that differ in image resolution ratios, quantities, crowding degree, and color spaces.Training uses pretrained Twins-SVT-large, random cropping and flipping, AdamW, and a batch size of 8.Source: Implementation Details · p. 4

Comparisons with State-of-the-art Methods

Ablation Study

  • Aggregating features from three transformer stages achieves better performance than using the last-stage feature alone.The last stage already outperforms most state-of-the-art methods, while other stages add information lost there.Source: Table 3 · p. 5, Ablation Study · p. 5

Visualization

  • Figure 3 visualizes CCTrans on five NWPU-Crowd scenes with varied scale, density, and illumination.The authors describe these visualizations as showing the model's fitting ability.Source: Visualization · p. 5, Figure 3 · p. 6

Ablation Study

  • Properly aggregating MDC columns improves performance, while stacking deeper convolution layers is not helpful for crowd counting.Stacking layers with varied dilation rates can address the adverse gridding effect; deeper stacking increases training time by 15% to 20%, whereas MDC adds less than 5%.Source: Ablation Study · pp. 5–6, Table 4 · p. 6, Table 5 · p. 6
  • Tailored losses lower fully supervised MAE by 2.5 on Part A and 2.1 on Part B, and smooth L1 lowers weakly supervised MAE by 0.7 and 1.3.The smooth L1 comparisons are against L1 on ShanghaiTech A and B, respectively.Source: Table 6 · p. 6, Ablation Study · p. 6
  • On ShanghaiTech, PFA and MDC contribute to final MAE, while Ld plays an important role in decreasing MSE.The component analysis evaluates the pipeline's components on ShanghaiTech.Source: Ablation Study · pp. 5–6, Table 7 · p. 6

Related work

Crowd-counting work moved from CNN-based detection and density estimation toward transformer backbones for global context. Prior transformer crowd counters use different designs, while CCTrans combines a pyramid transformer with a convolutional decoder.

CNN-based methods

  • CNN crowd-counting methods are broadly detection-based or density-based; density maps encode location probabilities whose pixel values sum to the crowd count.Detection methods predict person boxes but struggle with occlusion; RGBD-Net adds depth information but needs extra data and has limited performance in congested scenes.Source: CNN-based methods · p. 7
  • Density-based CNN methods address scale and density variation with multi-column networks, attention, auxiliary localization, specialized losses, or weakly supervised count regression.Examples include MCNN and Switch-CNN for scale variation, ASNet for density levels, and MATT for point-level weak supervision.Source: CNN-based methods · p. 7

Vision Transformer

  • Vision transformers use self-attention and pyramid designs for dense prediction, while crowd-counting studies include TransCrowd for weak supervision and BCCT for full supervision.CCTrans is described as a hybrid pyramid-transformer and convolutional-decoder framework that outperforms both methods across several benchmarks in both supervision settings.Source: Vision Transformer · p. 7

Conclusion

CCTrans is a four-component pipeline for weakly and fully supervised crowd counting, and the paper reports state-of-the-art results across several popular benchmarks. The authors propose it as a baseline for future work or for adaptation to other counting tasks.

Conclusion

  • CCTrans combines a pyramid vision transformer, feature aggregation, a multi-scale regression head, and tailored loss functions for both weakly and fully supervised crowd counting.The components target global context, coarse-to-fine information, multi-scale receptive fields, and training stability, respectively.Source: Conclusion · p. 7
  • The authors report that CCTrans advances the state of the art by clear margins across several popular benchmarks.Source: Conclusion · p. 7
  • The authors propose CCTrans as a strong baseline for further research or a pipeline that could be ported to other counting tasks.Source: Conclusion · p. 7

Dataset.

The evaluation spans five crowd-counting benchmarks with varied image properties and crowd conditions. The dataset descriptions include distinct data constraints and note dataset-specific training and validation settings.

Dataset

  • CCTrans is evaluated on UCF CC 50, ShanghaiTech Parts A and B, UCF QNRF, and NWPU-Crowd, which vary in resolution ratios, quantities, crowding, and color spaces.The paper says performance across these benchmarks demonstrates handling crowds under different situations.Source: Dataset · p. 10
  • For ShanghaiTech Part B and UCF QNRF, the experiments replace the OT loss with Bayesian loss; QNRF validation images are limited to 2400 pixels on the longer side.The reported validation results are described as the generally best results after 1500 epochs under the stated settings.Source: Dataset · p. 10
  • UCF CC 50 has 50 grayscale images, severe perspective distortion, and 63,974 head annotations, averaging 1,280 people per image.The dataset description notes little training data and no color information, while reporting strong CCTrans performance in crowded scenes.Source: Dataset · p. 10
  • UCF QNRF contains 1,535 images with 1,251,642 head annotations, split into 1,201 training and 334 testing images; most pictured objects are small in scale.Source: Dataset · p. 10
  • NWPU-Crowd contains 5,109 images and 2,133,375 annotated people, with variation in scale, density, and crowd count.The paper characterizes its distribution as closest to real crowds and says accurate results suggest CCTrans is suitable for real production.Source: Dataset · p. 10

Visualization

The paper visualizes CCTrans density maps across three datasets and reports that it handles images from different sources and color spaces. It also reports a No.1 NWPU-Crowd leaderboard ranking.

Visualization

  • CCTrans ranks No.1 on the NWPU-Crowd leaderboard.Source: Visualization · p. 10
  • The density-map visualization covers three datasets and is presented as evidence that CCTrans handles images from different sources and color spaces.Source: Visualization · p. 10
  • Figure 4 places original images in the left column, ground-truth density maps in the middle, and generated density maps on the right.Source: Figure 4 · p. 11

Related papers

Cited by

Loading 2109.14483v1…