PaperlayerView exact revision on arXiv

Source-linked AI summary

Counting with Focus for Free

Zenglin Shi, Pascal Mettes, Cees G. M. Snoek

arXiv:1903.12206v2cs.CV

TL;DR

Density-based counting commonly uses point annotations to construct density maps, leaving their other supervisory potential unused. This paper adds segmentation and global-density focus branches and a non-uniform kernel estimator, reporting reduced counting errors across six datasets. It also identifies extremely dense scenes as an open challenge.

  • Problem

    Density-based counting uses point annotations to construct density maps, although those points may provide additional supervision for counting networks.

    Source: §1 · p. 1

  • Method

    The method adds point-supervised segmentation and global-density focus branches, alongside a non-uniform kernel estimator for density maps.

    Source: §1 · pp. 1–2

  • Results

    Experiments on six datasets show reduced counting error regardless of base network, with reported state-of-the-art results on five benchmark datasets.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Segmentation and global-density focus complement each other by aiding density estimation from local and global perspectives.

    Source: §6 · p. 8

    Counting in extremely dense scenes remains an open problem, particularly when objects are hard to distinguish or blend with context.

    Source: §5.5 · p. 8, §6 · p. 8

Abstract

from arXiv · show

This paper aims to count arbitrary objects in images. The leading counting approaches start from point annotations per object from which they construct density maps. Then, their training objective transforms input images to density maps through deep convolutional networks. We posit that the point annotations serve more supervision purposes than just constructing density maps. We introduce ways to repurpose the points for free. First, we propose supervised focus from segmentation, where points are converted into binary maps. The binary maps are combined with a network branch and accompanying loss function to focus on areas of interest. Second, we propose supervised focus from global density, where the ratio of point annotations to image pixels is used in another branch to regularize the overall density estimation. To assist both the density estimation and the focus from segmentation, we also introduce an improved kernel size estimator for the point annotations. Experiments on six datasets show that all our contributions reduce the counting error, regardless of the base network, resulting in state-of-the-art accuracy using only a single network. Finally, we are the first to count on WIDER FACE, allowing us to show the benefits of our approach in handling varying object scales and crowding levels. Code is available at https://github.com/shizenglin/Counting-with-Focus-for-Free

1. Introduction

The paper argues that point annotations can supervise more than density-map construction, and repurposes them to guide counting through segmentation and global-density focus. It also introduces a kernel estimator and reports state-of-the-art accuracy on six datasets.

1. Introduction

  • Point annotations, usually used only to construct density maps, can provide additional supervision for optimizing deep-network counting.The paper frames this as reusing an existing source of supervision rather than requiring new annotations.Source: §1 · p. 1
  • Segmentation focus uses point-derived binary maps in a separate branch and loss to guide the network toward areas of interest.Global-density focus uses the relative amount of point annotations to regularize the predicted global image density.Source: §1 · p. 1
  • The paper introduces a kernel density estimator for non-uniform point distributions and an encoder-decoder network designed for varying object scales.The focus branches and density estimation are combined in a single network trained end-to-end with a multi-level loss.Source: §1 · pp. 1–2
  • Experiments on six datasets report state-of-the-art counting accuracy, and the paper presents the first counting results on WIDER FACE.The WIDER FACE results are used to demonstrate performance across varying object scales and crowding levels.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Prior counting work uses density-estimation networks, multi-task learning, and attention in different ways. This paper distinguishes its approach by repurposing point annotations as supervision for focus branches alongside density estimation.

Density-based counting

  • Multi-column networks target different object scales, while single-column networks offer architectural simplicity and training efficiency.The related work notes that multi-column networks can be difficult to train because of structural redundancy and optimization conflicts.Source: §2 · p. 2
  • Prior multi-task counting methods use classifier-selected regressors or helper networks, whereas this work adds supervised segmentation- and global-density-focus branches.The paper states that its focus is complementary to other state-of-the-art solutions and benefits counting regardless of the base network.Source: §2 · p. 2

Counting with attention

  • Soft attention learns weights for image regions or feature channels through a network branch, commonly as spatial or channel attention.The passage describes soft attention as differentiable and directly incorporable into an end-to-end network.Source: §2 · p. 2
  • Prior density-counting attention can fuse detection- and regression-based density maps, while this paper derives focus supervision from existing point annotations.In the cited prior approach, the attention map is implicitly learned with the density-map regression loss.Source: §2 · p. 2

3. Focus for Free

The method repurposes point annotations to supervise spatial and global-density focus, estimates point kernels locally, and combines both focus signals with a base network for density prediction.

3.1. Focus from segmentation

  • Point-derived binary segmentation supervision focuses the network on pixels near annotations, addressing the dominance of irrelevant pixels in the density loss.Pixels within the estimated variance range of any point annotation are labeled foreground.Source: §3.1 · p. 3

3.2. Focus from global density

  • Bilinear pooling captures global context to predict a channel-wise focus map that complements the spatial focus from segmentation.The global-density branch matches predicted global density to the ground-truth global density.Source: §3.2 · pp. 3–4
  • Patch-level global density is discretized from point counts relative to pixel counts, and its loss matches estimated and ground-truth density levels.The dataset step size is based on the maximum global density over image patches.Source: §3.2 · pp. 3–4

3.3. Non-uniform kernel estimation

  • The kernel estimator uses annotations in a point-centered local region, avoiding the assumption that points are uniformly distributed across the whole image.The region dimensions are set to one-eighth of the image size in the experiments.Source: §3.3 · p. 4

3.4. Architecture and optimization

  • An encoder-distiller-decoder aggregates encoder features from levels 4, 5, 7, and 8 to capture information useful for counting across object scales.The design omits level 6 because its large dilation rates can cause gridding artifacts.Source: §3.4 · pp. 4–5
  • The density branch multiplies base-network features by tiled segmentation and global-density focus maps before producing the output density map.The three branch losses are summed; density estimation jointly uses L1 and L2 losses, with weights (1, 10, 1) for the density, segmentation, and global-density losses.Source: §3.4 · p. 5

4. Experimental Setup

The experiments use several crowd, vehicle, cell, and face-counting datasets, with training and testing procedures specified for each. They report global and regional count errors, plus density-map quality on ShanghaiTech Part A.

4.1. Datasets

  • The datasets include ShanghaiTech Parts A and B, TRANCOS, DCC, UCF-QNRF, and WIDER FACE, whose validation set is used because test labels are unavailable.WIDER FACE provides face boxes, enabling kernel-estimator evaluation and analysis across object scales and crowding levels.Source: §4.1 · p. 5

4.2. Implementation details

  • Images are RGB-normalized by 255 and randomly cropped to 128 × 128 patches during training, while testing uses uncropped images.Training uses Adam with batch size 16 for up to 1000 epochs; kernel settings and global-density levels vary by dataset.Source: §4.2 · p. 5

4.3. Evaluation metrics

  • MAE and RMSE measure global count error, GAME aggregates error over local grids (with L=0 equivalent to MAE), while PSNR and SSIM assess density-map quality only on ShanghaiTech Part A.Source: §4.3 · pp. 5–6
  • On ShanghaiTech Part A and WIDER FACE, focus from segmentation outperforms the base network across object scales, even with spatial attention.The WIDER FACE comparison covers small, medium, and large scales.Source: Table 1 · p. 6

5. Results

The proposed approach improves counting across datasets, focus variants, and kernel estimation, with results compared against state-of-the-art methods. Its reported strengths include local counting of overlapping vehicles, while extremely dense or context-blended scenes remain challenging.

5.1. Focus from segmentation

  • On WIDER FACE, segmentation focus lowers MAE to 4.3 from 4.7 for the base network and 4.8 with spatial attention across object scales.On ShanghaiTech Part A, it reduces MAE from 74.8 to 72.3, while spatial attention raises it to 84.5.Source: §5.1 · p. 6

5.2. Focus from global density

  • On ShanghaiTech Part A, global-density focus lowers MAE to 71.7 from 74.8 for the base network and 73.4 and 72.6 for channel-attention variants.On WIDER FACE, it achieves its strongest result in sparse scenes.Source: §5.2 · p. 6, Table 2 · p. 6

5.3. Combined focus for free

  • Combined focus achieves MAE of 67.9 on ShanghaiTech Part A and 3.2 on WIDER FACE, outperforming combined-attention alternatives across datasets, scales, and crowding levels.It also lowers count error across the tested base networks on both datasets.Source: §5.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

5.4. Non-uniform kernel estimation

  • The proposed kernel lowers count error versus GAK on ShanghaiTech Part A and WIDER FACE, and reduces WIDER FACE variance-estimation MAE from 2.6 to 2.2.The variance comparison uses box annotations to obtain ground-truth variance.Source: Table 5 · p. 7, §5.4 · p. 7

5.5. Comparison to the state-of-the-art

  • For all TRANCOS GAME settings, the method sets a new state of the art, with a widening advantage as local-count penalties increase.The paper attributes this pattern to more accurate localization and counting of extremely overlapping vehicles.Source: §5.5 · p. 7, Figure 3 · p. 8
  • The proposed method sets a new state of the art on all six datasets for almost all global-count metrics.It achieves the best MAE and NMAE on WIDER FACE; Table 6 reports global-count comparisons across all six datasets.Source: §5.5 · p. 7, Table 6
  • The method directly learns full-resolution density maps, which the paper reports are higher quality than reduced-resolution maps recovered by interpolation.This comparison is made against methods such as [14].Source: §5.5 · pp. 7–8
  • Extremely dense scenes with indistinguishable objects, or objects blended with context, remain challenging failure cases.The authors identify these as open challenges.Source: §5.5 · p. 8

6. Conclusion

The paper repurposes point annotations for segmentation focus and global-density regularization, alongside a non-uniform kernel estimator. Its approach improves counting across object scales and crowding levels, while extremely dense scenes remain an open problem.

6. Conclusion

  • Segmentation focus directs the network to areas of interest, while global-density focus regularizes it to learn a matching global density.The paper describes these local and global forms of focus as complementary to density estimation.Source: §6 · p. 8
  • Experiments show benefits across object scales, crowding levels, and base networks, with state-of-the-art results on five benchmark datasets.The conclusion also describes WIDER FACE results elsewhere in the paper, but does not include that dataset among the five benchmarks stated here.Source: §6 · p. 8
  • Counting in extremely dense scenes remains an open problem, and the qualitative analysis identifies further improvement as necessary when objects are hard to distinguish.Figure 4 contrasts accurate counting when objects are individually visible with the need for improvement in dense settings.Source: §6 · p. 8, Figure 4 · p. 8

Appendix A

Appendix A presents the encoder-distiller-decoder ablation, evaluates non-uniform kernel estimation across counting networks, and provides qualitative examples of methods and outcomes.

Appendix A

  • The appendix covers network architecture and kernel ablations, qualitative effects of the focus methods, and success and failure cases across six datasets.Its stated scope includes the encoder-distiller-decoder network and non-uniform kernel estimation across counting networks.Source: Appendix A · p. 10

A.1. Encoder-Distiller-Decoder Network

The encoder-distiller-decoder combines multiscale feature fusion with detail-preserving decoding. The ablation reports its best counting results across all metrics when the distiller is placed between encoder and decoder.

A.1. Encoder-Distiller-Decoder Network

  • Adding the distiller to an encoder reduces MAE from 114.8 to 82.5 by fusing features from convolution layers with varying dilation rates.The paper says this helps count multiple objects appearing at multiple scales.Source: §A.1 · p. 10
  • A decoder preserves spatial resolution and important details better than an encoder alone or an encoder-distiller without decoding.The encoder-only network compresses objects and produces lower-resolution density maps; the distiller cannot recover resolution and detail as well as the decoder.Source: §A.1 · p. 10
  • The encoder-distiller-decoder achieves the best counting results on all metrics by combining scale invariance with detail-preserving density maps.Qualitative examples show lower count error and higher-quality density maps with less noise.Source: §A.1 · p. 10

A.2. Benefit of non-uniform kernel across counting networks

The proposed non-uniform kernel estimator is evaluated on ShanghaiTech Part A using the authors’ network and two alternative counting networks. It outperforms the geometry-adaptive kernel in MAE and RMSE for all three networks.

A.2. Benefit of non-uniform kernel across counting networks

  • The non-uniform kernel yields better MAE and RMSE than the geometry-adaptive kernel for all three evaluated counting networks.The evaluation includes the authors’ network and two alternatives, and the table compares density maps generated with the proposed kernel against GAK.Source: §A.2 · p. 10, Table 8 · p. 10

A.3. Qualitative Results for Segment-, Density- & Combined-Focus

Both segmentation focus and global-density focus suppress noise and reduce counting error; combining them yields the lowest error and higher-quality, less noisy density maps.

A.3. Qualitative Results for Segment-, Density- & Combined-Focus

  • Combining segmentation and global-density focus produces the lowest counting error and higher-quality density maps with less noise.Each focus alone also suppresses noise and reduces counting error compared with no focus.Source: §A.3 · p. 10

A.4. Success and Failure Cases

Qualitative examples show accurate counts in challenging scenes across six datasets, while scenes with extremely dense small objects remain a challenge.

A.4. Success and Failure Cases

  • Accurate counts appear in challenging scenes across six datasets, but extremely dense scenes with small objects remain a major challenge.The paper identifies these failure cases as an opportunity for future work.Source: §A.4 · p. 10
  • The encoder-distiller-decoder network integrates features from multiple encoder levels by attending to different image regions for a better overall representation.Figure 5 also visualizes the mean feature map from different levels.Source: Figure 5 · p. 11
  • Figure 6 contrasts density maps from the encoder, encoder-distiller, and encoder-distiller-decoder against ground truth on ShanghaiTech Part A.Source: Figure 6 · p. 11
  • Figure 7 compares density maps without focus, with segmentation focus, with global-density focus, and with combined focus against ground truth on ShanghaiTech Part A.Source: Figure 7 · p. 12
  • Figures 8–13 pair sample images with predicted density maps and ground truth for ShanghaiTech Part A and Part B, TRANCOS, DCC, UCF-QNRF, and WIDER FACE.Source: Figure 8 · p. 13, Figure 9 · p. 14, Figure 10 · p. 15
Loading 1903.12206v2…