PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning To Count Everything

Viresh Ranjan, Udbhav Sharma, Thu Nguyen, Minh Hoai

arXiv:2104.08391v1cs.CV

TL;DR

The paper addresses the limited category coverage and annotation cost of existing visual counting methods by formulating counting as few-shot regression. It introduces FamNet with exemplar-based test-time adaptation and the FSC-147 dataset, and reports that the approach outperforms several state-of-the-art detectors and few-shot counting methods.

  • Problem

    Existing visual counting methods mainly handle specific categories and require extensive object annotations, while suitable multi-category datasets for few-shot counting are lacking.

  • Method

    FamNet takes a query image and a few exemplars, predicts a density map using category-agnostic matching, and adapts at test time with a few gradient descent updates.

  • Results

    FamNet outperforms several state-of-the-art object detectors and few-shot counting approaches on the introduced few-shot counting setting.

  • Takeaways & Limitations

    Few-shot regression, exemplar-based adaptation, and FSC-147 provide a framework for evaluating counting across 147 visual categories with over 6000 images.

  • Takeaways & Limitations

    The dataset selection assumes that object instances have somewhat similar poses, texture, and appearance.

Abstract

from arXiv · show

Existing works on visual counting primarily focus on one specific category at a time, such as people, animals, and cells. In this paper, we are interested in counting everything, that is to count objects from any category given only a few annotated instances from that category. To this end, we pose counting as a few-shot regression task. To tackle this task, we present a novel method that takes a query image together with a few exemplar objects from the query image and predicts a density map for the presence of all objects of interest in the query image. We also present a novel adaptation strategy to adapt our network to any novel visual category at test time, using only a few exemplar objects from the novel category. We also introduce a dataset of 147 object categories containing over 6000 images that are suitable for the few-shot counting task. The images are annotated with two types of annotation, dots and bounding boxes, and they can be used for developing few-shot counting models. Experiments on this dataset shows that our method outperforms several state-of-the-art object detectors and few-shot counting approaches. Our code and dataset can be found at https://github.com/cvlab-stonybrook/LearningToCountEverything.

1. Introduction

The paper reframes visual counting as a few-shot regression problem for novel categories, introducing FamNet and FSC-147 to reduce annotation demands and support broad-category evaluation.

  • Motivation: Current counting networks typically handle one category at a time and require extensive labeled data, limiting scalability across visual categories.Existing systems map images to density maps, often requiring annotations for millions of objects across thousands of training images.
  • Problem formulation: Few-shot counting uses an image and a few bounding-box exemplars from that image to predict the count of similar object instances.The objective is to count all objects of a novel class represented by the exemplars.
  • Method: FamNet combines a general feature extractor with a category-agnostic density prediction module and adapts at test time using exemplar-driven gradient updates.The adaptation scheme uses two loss functions designed to exploit exemplar locations.
  • Dataset: FSC-147 provides more than 6000 images from 147 visual categories with dot and bounding-box annotations for few-shot counting research.The dataset was introduced to address the lack of a broad, unconstrained dataset for developing and evaluating general counting methods.
  • Contributions: The paper’s contributions are few-shot regression, FamNet with test-time adaptation, and the FSC-147 dataset.These contributions address both the annotation burden and the lack of a multi-category counting benchmark.

2. Related Works

Prior work largely targets category-specific counting or related few-shot tasks, while this paper distinguishes few-shot counting through exemplar matching, density estimation, and lightweight test-time adaptation.

  • Category-specific counting: Earlier counting methods commonly target people, cars, animals, cells, or fruits and often require tens of thousands or millions of annotated instances.Some methods reduce annotation costs by adapting a source-domain counting network to a target domain.
  • Matching-based counting: FamNet exploits similarity between a query image and exemplar objects, relating it to self-similarity methods and generic matching networks.GMN performs well with several dozen to hundreds of adaptation examples but does not perform well on novel classes without adaptation.
  • Few-shot counting versus detection: Few-shot counting differs from few-shot detection because it uses dot annotations and density estimation, which is described as more robust to occlusion than detection-then-counting.Density estimation avoids early binarized decisions and has been used in crowd and cell counting.
  • Few-shot adaptation: The proposed adaptation scheme draws on few-shot classification and performs a few gradient descent steps without requiring higher-order gradients during training.The paper reports better performance and faster training than MAML.

3. Few-Shot Adaptation & Matching Network

FamNet counts objects from a query image using a few exemplar bounding boxes, producing a density map whose summed values give the count. Its category-agnostic architecture is adapted at test time with exemplar-location losses.

  • Network architecture: FamNet takes a query image and a few exemplar bounding boxes as input, then predicts a density map whose pixel sum yields the object count.The density map has the same spatial size as the input image.
  • Network architecture: The network combines a multi-scale feature extraction module with a density prediction module designed to handle novel categories at test time.The feature extractor uses an ImageNet-pretrained network, while the density predictor is category agnostic.
  • Network architecture: Correlation maps between scaled exemplar features and image features are concatenated and fed to the density prediction module for category-agnostic prediction.The exemplar scales used are 0.9, 1.0, and 1.1, supporting objects at different scales.
  • Training: FamNet is trained with mean squared error between predicted and ground-truth density maps, using adaptive Gaussian smoothing to accommodate object-size variation.Adaptive smoothing estimates object size from nearest-neighbor distances between dot annotations.
  • Test-time adaptation: At test time, exemplar bounding-box locations guide adaptation through Min-Count and Perturbation losses that update the density prediction module.Min-Count requires at least one predicted density unit inside each exemplar box, while Perturbation encourages a Gaussian response around exemplar locations.
  • Test-time adaptation: The adaptation loss is used only during testing because the training loss over all pixels provides stronger supervision during training.The test-time procedure performs gradient-based updates using the combined adaptation loss.

4. The FSC-147 Dataset

FSC-147 is a diverse few-shot counting dataset created because existing datasets largely focus on single categories and lack suitable multi-instance images. It spans 147 categories with dot annotations for all instances and bounding boxes for exemplars, alongside disjoint category splits and benchmark comparisons.

  • Dataset motivation and scope: FSC-147 contains 6135 images across 147 object categories, covering items such as kitchen utensils, stationery, vehicles, and animals.The dataset was collected and annotated specifically for few-shot counting.
  • Annotations: Each image includes dot annotations for object centers, while three randomly selected exemplar instances also receive axis-aligned bounding boxes.The two annotation types support both counting supervision and few-shot exemplar specification.
  • Collection and filtering: The dataset was assembled by web image retrieval followed by manual filtering for quality, at least seven objects, appearance similarity, and limited severe occlusion.Candidate images were gathered using category-focused keyword searches across Flickr, Google, and Bing.
  • Dataset splits: Train, validation, and test sets use disjoint object categories, with 89 categories for training and 29 categories each for validation and testing.The splits contain 3659, 1286, and 1190 images, respectively.
  • Benchmark comparison: FamNet achieves the lowest MAE and RMSE on both validation and test sets among the methods compared in Table 1.The comparison includes Mean, Median, two few-shot detectors, GMN, and MAML.

5. Experiments

Experiments show FamNet is effective for few-shot counting across generic categories, outperforming competing approaches and pre-trained detectors. Ablations indicate that exemplar count and the model’s key components influence performance.

  • Evaluation Protocol: The evaluation uses MAE and RMSE to measure counting accuracy from predicted and ground-truth counts.These are standard counting metrics used to compare the methods.
  • Comparison with Few-Shot Approaches: FamNet outperforms all compared few-shot counting methods in Table 1.The comparison includes trivial average- and median-count baselines, few-shot detectors, GMN, and MAML.
  • Comparison with Object Detectors: FamNet outperforms pre-trained FasterRCNN, MaskRCNN, and RetinaNet detectors on categories represented in COCO.This holds even though those detectors were trained with thousands of annotated examples.
  • Ablation Studies: FamNet provides a reasonable count with one exemplar, outperforms competing methods with two, and improves as exemplars increase from one to three.The validation analysis varies the number of test-time exemplars between one and three.
  • Ablation Studies: Each analyzed FamNet component improves validation performance: multi-scale image features, multi-scale exemplar features, and test-time adaptation.The ablation removes these components individually or together on FSC-147 validation data.
  • Qualitative Results: FamNet’s qualitative failure case arises when background regions resemble the foreground object.A qualitative example also shows test-time adaptation decreasing density values in dense regions and improving the initial count.

6. Conclusions

The paper formulates visual counting as a few-shot regression task and presents a density-prediction approach with test-time adaptation. It also evaluates the approach against existing detectors and few-shot counting methods.

  • Counting is posed as a few-shot regression task for visual counting.
  • The paper collects a visual counting dataset with a relatively large number of object categories for the few-shot counting task.
  • The proposed approach predicts density maps and adapts at test time using exemplar instances from the target category.During adaptation, density values at dense locations decrease when the initial prediction overcounts.
  • The approach outperforms the state-of-the-art detectors and few-shot counting methods evaluated in the paper.
Loading 2104.08391v1…