PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Using Count-level Weak Supervision for Crowd Counting

Yinjie Lei, Yan Liu, Pingping Zhang, Lingqiao Liu

arXiv:2003.00164v1cs.CVcs.AI

TL;DR

Crowd counting commonly relies on location annotations that are costly to collect in dense scenes, while total counts can be easier to obtain. The paper trains with a small amount of location-level supervision and abundant count-level supervision, using MATT to constrain predicted density maps. It reports that MATT outperforms straightforward weakly supervised counting on a new dataset and traditional datasets.

  • Problem

    Location-level annotations are costly in crowded scenes, and it remains unclear how much training can benefit from abundant count-level annotations.

    Source: §1 · pp. 2–3

  • Method

    MATT trains with a small set of location-labeled images and many count-labeled images, using auxiliary branches and losses to constrain predicted density maps.

    Source: §1 · p. 3, §3.2 · p. 8

  • Results

    MATT outperforms straightforward weakly supervised crowd counting on a new dataset and traditional crowd-counting datasets.

    Source: §1 · p. 3, §6 · p. 24

  • Takeaways & Limitations

    Count-level weak supervision can help build a better crowd-counting model while alleviating annotation burden.

    Source: §1 · p. 3, §6 · p. 24

    The setting assumes a small set of location-labeled images alongside a large set of count-labeled images.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

Most existing crowd counting methods require object location-level annotation, i.e., placing a dot at the center of an object. While being simpler than the bounding-box or pixel-level annotation, obtaining this annotation is still labor-intensive and time-consuming especially for images with highly crowded scenes. On the other hand, weaker annotations that only know the total count of objects can be almost effortless in many practical scenarios. Thus, it is desirable to develop a learning method that can effectively train models from count-level annotations. To this end, this paper studies the problem of weakly-supervised crowd counting which learns a model from only a small amount of location-level annotations (fully-supervised) but a large amount of count-level annotations (weakly-supervised). To perform effective training in this scenario, we observe that the direct solution of regressing the integral of density map to the object count is not sufficient and it is beneficial to introduce stronger regularizations on the predicted density map of weakly-annotated images. We devise a simple-yet-effective training strategy, namely Multiple Auxiliary Tasks Training (MATT), to construct regularizes for restricting the freedom of the generated density maps. Through extensive experiments on existing datasets and a newly proposed dataset, we validate the effectiveness of the proposed weakly-supervised method and demonstrate its superior performance over existing solutions.

1. Introduction

The paper studies crowd counting with a small set of location-labeled images and many count-labeled images, aiming to reduce the burden of labor-intensive location annotation. It proposes MATT to constrain density maps on weakly annotated images and reports gains over a naive method on a new dataset and traditional datasets.

1. Introduction

  • Location-level dot annotations are simpler than box or pixel labels but remain labor-intensive and time-consuming in crowded scenes, while counts can be effortless to obtain in some practical scenarios.Images of the same objects can retain the same total count across spatial arrangements or viewpoints.Source: §1 · p. 2
  • MATT adds auxiliary prediction branches that impose count-matching and cross-branch consistency losses, encouraging features to encode location information for diverse density-map predictions.The paper argues that simply matching the integral of a predicted density map to the count does not sufficiently constrain maps on weakly annotated images.Source: §1 · p. 3
  • Experiments on a new dataset and traditional crowd-counting datasets show that MATT outperforms the straightforward weakly supervised method and leverages count-level supervision to build a better model.The new dataset is designed to test whether easily obtained count information benefits training.Source: §1 · p. 3
  • The paper identifies count-level weak supervision as a way to alleviate annotation burden and introduces a dataset for evaluating such methods.Its setting uses few location-labeled images and many count-labeled images.Source: §1 · pp. 3–4, Figure 1 · p. 4

2. Related Works

Prior crowd-counting work spans detection, feature and density regression, CNN-based density prediction, and lower-supervision approaches. The paper situates its contribution in count-level weak supervision, which it describes as not well explored.

2.1. Crowd Counting by Detection

  • Detection-based methods suit sparse regions but can fail in extreme cases such as low resolution and heavy occlusion.Earlier approaches also used hand-crafted features and detection of individual heads or body parts.Source: §2.1 · p. 4

2.2. Crowd Counting by Regression

  • Regression methods map image features to counts or density, with density-map integration providing the final crowd count.Earlier approaches included Gaussian Process and Random Forest regression.Source: §2.2 · pp. 4–5

2.3. Crowd Counting by CNN

  • CNN-based methods commonly predict density maps or counts with fully convolutional networks, and later work explored multi-scale, context, ensemble, attention, and other additions.The passage describes density-map regression as more robust than detection, segmentation, or direct count estimation.Source: §2.3 · p. 5

2.4. Crowd Counting by Low Supervision Methods

  • Most earlier methods assume location labels for all images; lower-supervision alternatives include unlabeled-data methods, count-supervised Gaussian processes, sparse-feature methods, and synthetic-image approaches.The paper identifies count-level weakly supervised crowd counting as not well explored and proposes a method to improve existing approaches.Source: §2.4 · pp. 5–6

3. Methodology

The methodology combines a small set of location-labeled images with a larger set of count-labeled images, then adds auxiliary density-map constraints and asymmetric training to address the weakness of count-only supervision.

3.1. Location-level Fully-supervised Crowd Counting

  • For location-level supervision, crowd-count targets are density maps formed by Gaussian smoothing annotated points, whose integral equals the object count.The model is trained to estimate these maps with a mean-square-error loss.Source: §3.1 · p. 7

3.2. Count-level Weak Supervision and Problem Formulation

  • The weakly supervised setting uses a small location-labeled image set and a large count-labeled set, since count-level annotations are weaker than location-level labels.The paper notes that seed-image counting can provide location annotations as a byproduct.Source: §3.2 · p. 8

3.3. Naive Solution for Weakly-supervised Crowd Counting

  • The naive solution adds an object-count loss to density-map training, but this weak constraint can yield inaccurate locations and poor generalization.The count loss encourages the density-map integral to match the known count.Source: §3.3 · pp. 8–9

3.4. Multiple Auxiliary Tasks Training (MATT)

  • MATT adds auxiliary branches whose density maps should represent the same dot map at different smoothness levels, using count and kernel-smoothed consistency losses.The branches share a feature extractor; the auxiliary branches are discarded at testing, leaving the extractor and primary branch.Source: §3.4 · pp. 9–10, Figure 2 · p. 10

3.5. Asymmetry Training

  • Asymmetric training fixes the primary branch when optimizing auxiliary consistency, so auxiliary branches learn from the primary branch but not vice versa.This is intended to prevent unreliable auxiliary predictions from altering primary-task training.Source: §3.5 · pp. 11–12

4. Multi-Shot Crowd Counting (MSCC) dataset

The MSCC dataset was created to test whether count information collected through the described image-sampling strategy can benefit training and whether the proposed method works on those samples. It covers 15 object categories and uses count-level annotations inferred from a manually counted seed image.

4. Multi-Shot Crowd Counting (MSCC) dataset

  • The dataset tests whether samples collected using the Section 3.2 strategy can benefit network training and whether the proposed method works on those samples.Figure 3 shows typical images from the 15 categories.Source: §4 · pp. 12–13, Figure 3 · p. 13
  • Each of 15 categories has 200 images across ten quantity levels, with counts inferred from a location-annotated seed image and subsequent object additions or removals.Images also vary in viewpoint, object layout, and background; Figure 4 illustrates the collection procedure.Source: §4 · pp. 13–14, Figure 4 · p. 13
  • Per category, training uses 169 count-level images and one location-level image, validation uses 30 count-level images, and testing uses 100 additional manually counted images.Test counts are designed to be more diverse than training counts to assess generalization to different object counts.Source: §4 · p. 14

5.1. Experimental Setting

Experiments compare three training approaches under a count-level weak-supervision setting, using MAE and MSE on traditional datasets and MAE and RER on MSCC. At test time, only the feature extractor and primary branch produce the final density maps.

5.1. Experimental Setting

  • Traditional datasets are evaluated with MAE and MSE, while MSCC uses MAE and RER, which divides MAE by total object count to express relative error.RER reflects that a larger absolute error may be more acceptable for a more crowded scene.Source: §5.1 · pp. 14–15
  • The three compared approaches are location-level-only Baseline1, count-level-integral Baseline2, and MATT with multiple auxiliary tasks.Table 2 compares the three methods; Baseline2 jointly trains on location-level density maps and count-level total counts.Source: §5.1 · p. 15, Table 2 · p. 16
  • At test time, auxiliary branches g1 through gk are discarded, and only fb and g0 generate the final density maps.This separates the auxiliary-task training setup from the branches used for final prediction.Source: §5.1 · p. 16

5.2. Implementation Details

The implementation uses a CSRNet-based feature extractor with one primary branch and four auxiliary branches. Training uses Adam, with hyperparameters selected on the validation set.

5.2. Implementation Details

  • The feature extractor uses CSRNet layers through its third-last layer as a 256-channel backbone, and the method can extend to other feature-extractor architectures.CSRNet is identified as a state-of-the-art crowd-counting solution.Source: §5.2 · p. 16
  • One primary branch and four auxiliary branches each use three 3 × 3 convolutional layers with channels reduced from 256 to 128, 64, and 1.Different predefined Gaussian kernels with different covariance matrices construct the auxiliary losses.Source: §5.2 · p. 16
  • The implementation uses PyTorch and Adam, with all hyperparameters chosen using the validation set.Source: §5.2 · p. 16

5.3. Datasets and Results

MATT improves on weakly supervised baselines across the reported datasets, with gains on ShanghaiTech, UCF CC 50, WorldExpo’10, and the newly introduced MSCC dataset. The experiments use few location-labeled images and many count-labeled images.

5.3.1. Evaluation on the ShanghaiTech dataset

  • The ShanghaiTech evaluation uses 270 count-labeled and 30 location-labeled training images in Part A, and 380 count-labeled and 20 location-labeled images in Part B.The section also lists fully supervised state-of-the-art results for reference; those are not directly comparable to the weakly supervised methods.Source: §5.3.1 · p. 17, §5.3 · p. 17

5.3.2. Evaluation on the UCF CC 50 Dataset

  • On UCF CC 50, MATT improves MAE by almost 50 heads over the naive method.The weakly supervised setup uses five location-labeled images and 45 count-labeled images.Source: §5.3.2 · p. 19, Table 4 · p. 19

5.3.3. Evaluation on the WorldExpo’10 Dataset

  • On WorldExpo’10, MATT achieves the lowest MAE in four of five scenarios and improves average MAE by more than 2.4 heads over Baseline1.The setup uses 169 location-labeled frames and 3,211 count-labeled frames.Source: §5.3.3 · pp. 19–20, Table 5 · p. 20

5.3.4. Evaluation on the MSCC Dataset

  • On MSCC, MATT averages MAE 6.8 and RER 15.7, respectively 63.8% and 76.4% lower than Baseline1.It achieves the best MAE and RER across all categories; for pin headers, MAE falls from 27.9 to 5.8 and RER is 87.8% lower.Source: §5.3.4 · p. 20, Table 6 · p. 21

5.4. Ablation Study

The ablations find that combining the auxiliary loss terms and using several auxiliary branches perform best in the tested settings. They also support asymmetric training and indicate that MATT’s gains are largely tied to count-level data.

5.4.1. Other Alternative Auxiliary Loss Functions

  • Combining the count and density-consistency loss terms performs best; the MSE component is reported as the most effective part of the total loss.The variation using only the first term in Eq. 4 is comparable to MATT, while the count-loss-only variation performs worse.Source: §5.4.1 · p. 21, Table 7 · p. 22

5.4.2. Varying the Number of Auxiliary Branches

  • On ShanghaiTech Part A, four to six auxiliary branches give the best performance; two branches reach MAE around 90.8 heads, no better than the naive method.The authors relate the benefit of more branches to maintaining diverse auxiliary supervision signals.Source: §5.4.2 · p. 22, Figure 6 · p. 22

5.4.3. The Importance of the Asymmetry Training

  • Removing asymmetric training yields MAE 89.0 and MSE 140.6 on ShanghaiTech Part A, substantially worse than the original method.The authors argue that auxiliary-task gradients could be noisy because auxiliary branches fit density maps predicted by the primary branch.Source: §5.4.3 · pp. 22–23, Table 8 · p. 23

5.4.4. MATT for Fully Annotated Data

  • On fully annotated ShanghaiTech Part A training data, MATT performs only slightly better than the single-task Baseline1.The authors conclude that MATT’s stronger performance largely comes from exploiting count-level images, not merely from multi-task learning.Source: §5.4.4 · p. 23, Table 9 · p. 24

6. Conclusions

The paper concludes that MATT outperforms straightforward weakly supervised crowd counting and leverages count-level supervision to build a better model.

6. Conclusions

  • Experiments on a new dataset and traditional crowd-counting datasets show MATT outperforming the straightforward weakly supervised method and leveraging count-level supervision to build a better crowd-counting model.Source: §6 · p. 24

Related papers

Loading 2003.00164v1…