PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-shot Object Counting with Similarity-Aware Feature Enhancement

Zhiyuan You, Kai Yang, Wenhan Luo, Xin Lu, Lei Cui, Xinyi Le

arXiv:2201.08959v5cs.CV

TL;DR

Few-shot object counting aims to count exemplar objects specified by support images, including when targets are densely packed. SAFECount compares projected support and query features to guide feature enhancement, and reports improvements over state-of-the-art methods on FSC-147 and CARPK. On FSC-147 test data, it surpasses FamNet+ by 7.76 MAE and 14.00 RMSE without test-time adaptation.

  • Problem

    Conventional counting methods often assume target classes appeared during training, while densely packed targets make individual objects hard to distinguish.

    Source: §1 · pp. 1–2

  • Method

    SAFECount compares projected support and query features, normalizes scores across exemplar and spatial dimensions, and uses the resulting similarities to weight support-feature fusion into query features.

    Source: §1 · p. 2, §3.2 · pp. 3–4

  • Results

    SAFECount substantially improves over state-of-the-art methods on FSC-147 and CARPK; on FSC-147 test data, it surpasses FamNet+ by 7.76 MAE and 14.00 RMSE without test-time adaptation.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Visualizations indicate that SAFECount benefits from clearer learned boundaries between densely packed objects.

    Source: §1 · p. 2

Abstract

from arXiv · show

This work studies the problem of few-shot object counting, which counts the number of exemplar objects (i.e., described by one or several support images) occurring in the query image. The major challenge lies in that the target objects can be densely packed in the query image, making it hard to recognize every single one. To tackle the obstacle, we propose a novel learning block, equipped with a similarity comparison module and a feature enhancement module. Concretely, given a support image and a query image, we first derive a score map by comparing their projected features at every spatial position. The score maps regarding all support images are collected together and normalized across both the exemplar dimension and the spatial dimensions, producing a reliable similarity map. We then enhance the query feature with the support features by employing the developed point-wise similarities as the weighting coefficients. Such a design encourages the model to inspect the query image by focusing more on the regions akin to the support images, leading to much clearer boundaries between different objects. Extensive experiments on various benchmarks and training setups suggest that we surpass the state-of-the-art methods by a sufficiently large margin. For instance, on a recent large-scale FSC-147 dataset, we surpass the state-of-the-art method by improving the mean absolute error from 22.08 to 14.32 (35%$\uparrow$). Code has been released in https://github.com/zhiyuanyou/SAFECount.

1. Introduction

Few-shot object counting lets users specify target objects with support images, addressing the limits of models tied to training classes, but densely packed objects remain difficult to distinguish. SAFECount combines similarity guidance with feature enhancement and reports improvements over state-of-the-art methods on FSC-147 and CARPK.

1. Introduction

  • Few-shot counting lets users specify target objects with support images, enabling one model to count varied objects and adapt to novel classes without retraining.The introduction contrasts this with methods that assume the test object class appeared in training.Source: §1 · p. 1
  • Earlier feature-based methods pool support features, while similarity-based methods regress from similarity maps that can lack clear boundaries for densely packed objects.Figure 2 contrasts these designs with SAFECount, which uses similarity to guide feature enhancement.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • SAFECount uses projected support-query comparisons to create normalized similarity maps, then uses point-wise similarities to fuse support features into query features.The similarity comparison module (SCM) normalizes across exemplar and spatial dimensions; the feature enhancement module (FEM) uses similarities as weighting coefficients.Source: §1 · p. 2
  • SAFECount improves over state-of-the-art methods on FSC-147 and CARPK, and benefits from clearer learned boundaries between densely packed objects.The paper reports that these boundaries help distinguish individual objects in dense query regions.Source: §1 · p. 2

2. Related Work

Prior counting methods commonly target classes represented in training, whereas few-shot object counting uses support images to specify targets, including novel classes. Existing FSC approaches compare support and query features or similarities, and SAFECount is positioned as integrating the advantages of both.

2. Related Work

  • Class-specific counting methods handle particular categories, while surveyed multiclass methods still cannot count classes beyond their training data.Examples include person counting and approaches based on region partitioning, segmentation, detection, or category attention.Source: §2 · p. 2
  • Few-shot object counting allows users to specify targets with one or several support images, including classes unseen during training.The paper frames this setting as a way to adapt a learned model to novel test classes using a few support samples.Source: §2 · pp. 2–3
  • FSC methods compare support and query features: GMN concatenates pooled support features, CFOCNet regresses from similarity maps, and FamNet adds multi-scale augmentation and test-time adaptation.The paper positions SAFECount as integrating features and similarities; similarity-map regression can fail to separate densely packed objects.Source: §2 · p. 3

3. Method

FSC uses a few support images to count objects from novel classes by leveraging knowledge learned from base classes. SAFECount enhances query features with support-guided similarity while preserving spatial structure, then regresses a density map.

3.1. Preliminaries

  • FSC counts exemplar objects from novel classes using a few support images and generalization knowledge learned from base classes.The setting is called K-shot FSC when each query has K support images.Source: §3.1 · p. 3

3.2. Similarity-Aware Feature Enhancement

  • SCM projects support and query features into a comparison space, compares them spatially, and normalizes scores across exemplar and spatial dimensions.The projection uses a 1×1 convolution and shared layer normalization; the score map is normalized to prevent unusually large values from dominating learning.Source: §3.2 · pp. 3–4
  • FEM uses similarity values to weight support features before integrating them into query features, preserving spatial structure associated with clearer object boundaries.Unlike attention modules that flatten feature maps, SAFECount maintains their spatial structure throughout SCM and FEM.Source: §3.2 · pp. 4–5

3.3. Training Framework

  • A fixed ImageNet-pretrained ResNet-18 and ROI pooling form the input features; an MSE-trained regression head maps enhanced query features to density maps, and blocks can be stacked.The paper reports that adding one block is already enough to boost performance substantially.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate SAFECount on FSC-147 under standard and cross-validation settings, test transfer to CARPK, and examine its design choices. Results show improvements over comparison methods, while ablations and visualizations probe the modules and object-boundary behavior.

4.1. Metrics and Datasets

  • Evaluation uses MAE and RMSE on FSC-147, a 147-class, 3-shot dataset with disjoint training, validation, and test classes, plus four-fold and 1-shot evaluations.CARPK supplies a separate car-counting transfer setting; its 1,448 images cover four parking-lot scenes.Source: §4.1 · p. 5, Table 1 · p. 6

4.2. Class-agnostic Few-shot Object Counting

  • SAFECount surpasses FamNet+ [24] on FSC-147 by 8.47 MAE and 21.87 RMSE on validation, and 7.76 MAE and 14.00 RMSE on test, without test-time adaptation.Qualitative examples compare ground truth, FamNet+ predictions, and SAFECount predictions, including densely packed objects.Source: §4.2 · p. 6, Table 2 · p. 6, Figure 4 · p. 7
  • SAFECount outperforms FamNet [24] across four FSC-147 folds and both shot settings, by an average 8.82 MAE in 1-shot and 9.67 MAE in 3-shot.The reported results also show larger gains from 1-shot to 3-shot than the two baseline methods.Source: §4.2 · p. 6, Table 3 · p. 7

4.3. Cross-dataset Generalization

  • On CARPK, SAFECount surpasses FamNet [24] by 42.23% in MAE and 45.85% in RMSE after FSC-147 pretraining, and GMN [20] by 28.74% in MAE and 28.89% in RMSE after fine-tuning.FSC-147 pretraining excludes the car category; the cross-dataset evaluation uses CARPK.Source: §4.3 · p. 6, Table 4 · p. 7

4.4. Ablation Study

  • Replacing FEM drops test MAE by 9.05, removing SCM by 1.81; ENorm and SNorm each improve test performance by at least 4 MAE, and together achieve the best result.Ablations also report that vanilla attention performs worse than the proposed modules.Source: §4.4 · pp. 6–7, Table 5 · p. 8
  • Four SAFECount blocks are selected for FSC-147: performance improves through four blocks and drops slightly at five; enhanced-feature regression beats raw or block similarity, and freezing the backbone surpasses training it.The authors attribute the enhanced-feature result partly to filtering erroneous high similarities, and suggest training may overfit to training classes.Source: §4.4 · pp. 7–8, Table 5 · p. 8
  • Kernel flipping improves performance by at least 1 MAE, with the authors linking the gain to preserving the spatial structure of the similarity-weighted feature.The ablation table includes the kernel-flipping comparison.Source: §4.4 · p. 8, Table 5 · p. 8

4.5. Visualization

  • SAFECount’s learnable similarity comparison and similarity-weighted support features retain image semantics while encoding support-query relations, yielding clearer boundaries between densely packed objects.The paper contrasts these visualized similarity maps with FamNet’s direct comparison of raw features.Source: §4.5 · p. 8, Figure 5 · p. 8

5. Conclusion

SAFECount combines similarity comparison with feature enhancement for few-shot object counting. Its normalized similarities guide support-feature integration, and experiments report state-of-the-art performance across benchmarks and training settings.

5. Conclusion

  • SAFECount combines a similarity comparison module (SCM) and a feature enhancement module (FEM) for few-shot object counting.SCM compares support and query features, while FEM uses the resulting similarity values to integrate support features into the query feature.Source: §5 · p. 8
  • Normalizing score maps across exemplar and spatial dimensions produces a similarity map that weights support-feature integration into the query feature.The design directs attention toward regions similar to the support images and brings distinguishable borders within densely packed objects.Source: §5 · p. 8
  • Experiments across benchmarks and training settings report state-of-the-art performance by a considerably large margin.Source: §5 · p. 8

A. Network Architecture and Training Configurations

SAFECount extracts query and support features, compares them in SCM, then uses FEM to enhance the query feature before density-map regression. The section also specifies the backbone and FSC-147 training configuration.

A. Network Architecture and Training Configurations

  • A frozen ImageNet-pretrained ResNet-18 supplies resized outputs from its first three residual stages, which are concatenated and reduced to a 256-channel query feature.The 1 × 1 reduction convolution is trainable; ROI pooling produces a 3 × 3 support feature for each support image.Source: A. Network Architecture and Training… · p. 10
  • SCM projects support and query features, compares them by convolving the query feature with support features as kernels, and outputs a similarity map.The score map is normalized across exemplar and spatial dimensions; Figure 6 illustrates feature comparison in the 1-shot case.Source: A. Network Architecture and Training… · p. 10, Figure 6 · p. 11
  • FEM uses similarity values as weights to aggregate support features, then applies learnable feature fusion to produce the enhanced query feature.Support features are flipped horizontally and vertically before convolution so the aggregated feature inherits their spatial structure; Figure 7 illustrates this operation.Source: A. Network Architecture and Training… · p. 11, Figure 7 · p. 11
  • The regression head maps the enhanced feature to a density map, and successive SAFECount blocks may use the preceding enhanced feature as their query input.For FSC-147 training, the model uses four blocks, batch size 8, and 200 epochs; Adam's epsilon is 4e-11 and the learning rate starts at 2e-5.Source: A. Network Architecture and Training… · pp. 11–12

B. Ablation Studies

The ablations examine the loss and ROI-pooling size. They retain MSE without the SSIM term and select 3 × 3 ROI pooling for FSC-147.

B. Ablation Studies

  • Adding SSIM improves MAE but worsens RMSE, so the authors omit it because the RMSE drop is too large.They speculate that SSIM may help some samples while harming samples with extremely large counts; Table 7 covers the loss-weight ablation.Source: B. Ablation Studies · p. 12
  • A 3 × 3 ROI-pooling size brings stable improvement over 1 × 1, while 5 × 5 slightly decreases performance, leading to the 3 × 3 choice for FSC-147.The authors attribute the 1 × 1 result to loss of support spatial information and suggest that 5 × 5 may hinder accurate localization; Table 7 covers the size ablation.Source: B. Ablation Studies · p. 12

C. More Results

This section presents additional results, including quantitative evaluation on class-specific counting datasets and visual samples.

C. More Results

C.1. Class-specific Object Counting

SAFECount is evaluated on five class-specific car and crowd datasets. It outperforms multi-class and few-shot counting baselines in several settings, with performance comparable to specialized single-class methods in others.

C.1. Class-specific Object Counting

  • The evaluation covers CARPK and PUCPR+ for car counting, plus UCSD, Mall, and ShanghaiTech Parts A and B for crowd counting.Five support images are randomly sampled from the training set and fixed for both training and test; the support feature size is 1 × 1 and the block number is 2.Source: §C.1 · p. 12, Table 8 · p. 12
  • SAFECount surpasses multi-class and few-shot counting methods on the car datasets, matches single-class car methods comparably, and shows dataset-specific advantages on crowd counting.It beats all counterparts on UCSD and Mall, and outperforms multi-class and few-shot methods on ShanghaiTech while matching specialized crowd methods competitively.Source: §C.1 · pp. 12–13, Table 9 · p. 13

C.2. More Qualitative Results

Qualitative examples show SAFECount handling unseen FSC-147 classes across varied object densities, scales, and shapes. Examples on car and crowd datasets also illustrate localization under occlusion and across crowd-density conditions.

C.2. More Qualitative Results

  • On unseen FSC-147 classes, SAFECount counts objects across high and sparse densities, varied scales and shapes, with high localization accuracy.Examples include dense Legos, sparse Prawn Crackers, small Birds, large Horses, round Apples, square Stamps, and strip-shaped Skis and Shirts.Source: §C.2 · p. 13, Figure 8 · p. 14, Figure 9 · p. 15
  • Car examples show successful localization across angles and scales, including cars in deep shadows or partly hidden under trees.The reported examples come from CARPK and PUCPR+.Source: §C.2 · p. 13
  • Crowd examples cover sparse UCSD and Mall scenes, distinguishable ShanghaiTech crowds, and denser Part A scenes where the model predicts density estimates.For ShanghaiTech Part B, where most people are distinguishable, the method localizes and counts them.Source: §C.2 · p. 13
Loading 2201.08959v5…