Source-linked AI summary
Class-Agnostic Counting
Erika Lu, Weidi Xie, Andrew Zisserman
TL;DR
Existing counting methods are usually class-specific, whereas this paper seeks flexible counting for any object specified by an exemplar. It recasts counting as matching, trains GMN with tracking data, and adapts it efficiently; the model is competitive across cells, cars, and crowds and strongly outperforms prior car-counting methods.
Problem
Existing counting methods are generally designed for specific object classes, motivating a model that can flexibly count instances specified by an exemplar patch.
Method
The Generic Matching Network recasts counting as matching self-similar exemplar instances, trains on tracking video data, and uses adapters for few-shot domain specialization.
Results
The model achieves competitive cell and crowd-counting performance and surpasses prior car-counting methods using only 3 training images, with larger gains when trained on the full dataset.
Takeaways & Limitations
GMN provides a flexible counting framework that can handle overlapping and well-separated objects and adapt across domains with limited labeled data.
Takeaways & Limitations
The current work counts individual images or frames; future work is needed to exploit within-frame and between-frame similarities in video sequences.
Abstract
from arXiv · showhide
Nearly all existing counting methods are designed for a specific object class. Our work, however, aims to create a counting model able to count any class of object. To achieve this goal, we formulate counting as a matching problem, enabling us to exploit the image self-similarity property that naturally exists in object counting problems. We make the following three contributions: first, a Generic Matching Network (GMN) architecture that can potentially count any object in a class-agnostic manner; second, by reformulating the counting problem as one of matching objects, we can take advantage of the abundance of video data labeled for tracking, which contains natural repetitions suitable for training a counting model. Such data enables us to train the GMN. Third, to customize the GMN to different user requirements, an adapter module is used to specialize the model with minimal effort, i.e. using a few labeled examples, and adapting only a small fraction of the trained parameters. This is a form of few-shot learning, which is practical for domains where labels are limited due to requiring expert knowledge (e.g. microbiology). We demonstrate the flexibility of our method on a diverse set of existing counting benchmarks: specifically cells, cars, and human crowds. The model achieves competitive performance on cell and crowd counting datasets, and surpasses the state-of-the-art on the car dataset using only three training images. When training on the entire dataset, the proposed method outperforms all previous methods by a large margin.
1 Introduction
The paper proposes class-agnostic counting by specifying an exemplar patch and matching repeated instances, using image self-similarity and tracking video data. An adapter supports specialization to new domains or user requirements with few labeled examples.
- Motivation: Existing counting methods generally target specific object classes through detection- or regression-based approaches.Detection-based methods require separate detectors and struggle with limited annotations, while regression-based methods estimate counts or density maps.
- Motivation: The proposed model counts instances by specifying an exemplar patch and exploiting repeated visual patterns within the image.Image self-similarity motivates recasting counting as matching.
- Approach: The Generic Matching Network learns to match exemplar instances and is trained using video data labeled for tracking.Tracking videos contain natural object repetitions under varying viewing conditions.
- Adaptation: An adapter module specializes matching to restrictive or broader user requirements using very few annotated examples and a small number of tunable parameters.Examples include counting red cars or cars across orientations.
- Results: The model adapts across cells, cars, and crowds, with competitive cell and crowd results and strong car performance using as few as 3 images.The introduction reports large-margin comparisons when training on the entire dataset.
2 Method
GMN embeds an exemplar and a full image, matches them densely to produce a similarity heatmap, and converts that map into counts or detections. It is pretrained on tracking data and adapted to new domains with a small trainable parameter subset.
- 2 Method: Counting is reformulated as locating and counting repeated instances by matching an exemplar patch within the image.This formulation supports class-agnostic counting because the desired object is specified by the exemplar.
- 2 Method: The GMN uses embedding, matching, and adapting modules to compare an exemplar with a full-resolution image.The architecture obtains counts and detections from the resulting output heatmap.
- 2.1 Embedding: The embedding module maps the exemplar to a feature vector and the full image to a dense feature map using separate functions.The exemplar patch and image are encoded with two streams; the described implementation uses truncated ResNet-50 networks.
- 2.2 Matching: The matching module broadcasts the exemplar vector, concatenates it with the image feature map, and predicts a dense similarity heatmap.A trainable function models relations between the exemplar and all image locations, with upsampling provided by a convolutional transpose layer.
- 2.3 Training Generic Matching Networks: The generic matching network is trained on ILSVRC tracking data with Gaussian instance labels and weighted MSE loss.Summing the output similarity map can therefore provide a count because the map also functions as a density map.
- 2.3 Training Generic Matching Networks: The model can match objects across viewpoint, occlusion, clothing, overlap, and substantial HeLa-cell appearance variation before target-domain adaptation.These observations support its use as an initialization for unseen domains.
- Adaptation: Adaptation freezes the pretrained network and trains residual adapters plus batch-normalization layers to change responses for new domains or requirements.The method adds adapters in parallel with existing convolutions, while counting must also handle noise, false positives, and difficult appearance changes.
3 Counting Benchmark Experiments
The GMN is adapted to cell and car counting benchmarks, handling both overlapping and well-separated instances. It remains competitive with very limited training data and achieves strong CARPK performance.
- Benchmark setup: The GMN is adapted to synthetic cells, HeLa cells, and drone-collected cars, each presenting distinct overlap, appearance, or scene challenges.Synthetic cells contain overlapping instances; HeLa cells have substantial variation with only 11 training images; CARPK contains varied orientations, occlusions, and distractors.
- Synthetic fluorescence microscopy: The synthetic-cell experiments show that GMN counting is not restricted to detection-based counting and performs well under high instance overlap.The comparison includes methods highly engineered for this dataset.
- HeLa cells: 11 training images support HeLa results comparable to conventional handcrafted-feature methods despite large intra-class variation in shape and size.The dataset has 11 training and 11 testing images, and MAE is calculated from detection counts because instances are well separated.
- Cars: 3 training images containing 99 cars produce state-of-the-art CARPK results, while full-dataset training further boosts performance by a large margin.The two GMN models are trained with augmentation on either three randomly sampled images or the full training set.
- Cars: MAE=7.5 is the best reported counting error, while the full-data model at T = 2.75 achieves MAE=23.8 versus MAE=19.7 for the previous state-of-the-art.Lowering the threshold increases recall at the expense of precision; the table evaluates MAE, RMSE, recall, and precision.
- Discussion: GMN supports integral-based counting for overlapping objects and local-maximum counting for separated objects, with an ensemble also possible.The same model handles density-estimation and detection-based scenarios, unlike approaches designed separately for each setting.
4 Shanghaitech Crowd Counting
The paper extends counting-by-matching to crowded human images by matching patch statistics rather than individual people. Preliminary Shanghaitech experiments support the approach’s potential for low-shot, category-agnostic crowd counting.
- 4 Shanghaitech Crowd Counting: Shanghaitech Part A contains very large crowds viewed from arbitrary perspectives, with people appearing at highly varied scales due to perspective.These conditions motivate a flexible matching-based formulation.
- 4 Shanghaitech Crowd Counting: The method conjectures that patches containing different numbers of people have distinguishable statistics, enabling crowd counting without matching individual instances.A patch with 10 people is expected to differ statistically from one with 20 people.
- 4 Shanghaitech Crowd Counting: 64 × 64 patches are quantized into 10 classes by people count, then embedded so same-class pairs match and different-class pairs do not.The model uses ground-truth dot annotations for quantization and trains a self-similarity architecture.
- 4 Shanghaitech Crowd Counting: Preliminary experiments demonstrate the possibility of scaling counting-by-matching to human crowd datasets while targeting low-shot category-agnostic counting.The comparison is against models specifically designed for human crowd counting.
5 Conclusion
The paper recasts counting as matching to use tracking video data and enable flexible domain adaptation. Future work includes video-sequence counting and improved scale-invariant feature fusion.
- 5 Conclusion: Recasting counting as matching enables use of object-detection video data and flexible adaptation across domains through few-shot learning.The authors present this restructuring as a step toward an all-purpose counting model.
- 5 Conclusion: Future work could extend the method from individual images or frames to video sequences, exploiting both within-frame and between-frame similarities.The authors identify the tracking analogue as especially significant in this setting.