CounTR: Transformer-based Generalised Visual Counting

• Human judgments are rapid, accurate, and confident mainly below five objects; as counts rise, accuracy and confidence decline, leaving estimation or enumeration slower or less accurate.The below-five ability is termed subitizing.Source: §1 · p. 1
• Generalised visual counting identifies salient objects from arbitrary semantic classes in an image, with any number of user-provided exemplars, from zero-shot to few-shot.The predicted density map is summed to obtain the final count.Source: §1 · pp. 1–2
• CounTR uses transformer attention to capture similarity among image patches and between patches and user-provided exemplars, then sums a predicted density map for the count.The architecture accepts few-shot annotations when provided.Source: §1 · p. 2
• The model is pre-trained with self-supervised masked image modeling and then supervised fine-tuned, with pre-training reported to significantly improve counting performance.The training scheme is intended to learn visual representations for counting.Source: §1 · p. 2
• A simple, scalable synthesis pipeline creates training images with many instances to address datasets’ long-tailed scarcity of high-count images and improve performance on such images.The pipeline also conditions training on user-provided instance exemplars.Source: §1 · p. 2
• CounTR achieves state-of-the-art zero-shot and few-shot performance on large-scale counting benchmarks, improving the previous best approach by over 18.3% on test-set mean absolute error.The introduction names FSC-147 as an example benchmark.Source: §1 · p. 2

Visual object counting

• Detection-based methods [13] localize instances, while regression-based methods map global or dense image features to counts or density maps.Regression handles overlapping instances better, but both approaches have been limited to counting one particular class; detection also requires class-specific detectors and is challenging with few annotations.Source: §2 · p. 2

Class-agnostic object counting

• Class-agnostic few-shot counting [19] [24] [30] uses exemplar instances to count objects across classes and requires models to learn cross-class commonalities.GMN [19] matches image and exemplar CNN features to regress density; FamNet [24] uses feature correlation and test-time adaptation; SAFECount [30] refines query features with support features before density regression.Source: §2 · pp. 2–3
• DINO with a lightweight regression head counts without exemplars, whereas this paper trains a transformer from scratch to count with any shot count.The cited class-agnostic approaches include GMN [19], FamNet [24], and SAFECount [30].Source: §2 · p. 3

3 Methods

• The task is to count salient objects from arbitrary semantic categories using zero or few exemplars, with training and test categories disjoint.The predicted object count is obtained by summing the spatial density map.Source: §3 · p. 3

3.1 Architecture

• CounTR encodes the query image and exemplars separately, interacts their features, and decodes the result into a density map whose sum gives the count.Without exemplars, a learnable [SPE] token supplies key and value features; the model should accommodate exemplar diversity while counting the specified semantic instances.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.1.1 Visual Encoder

• The image encoder splits inputs into 16×16 patches and processes their tokens with 12 ViT layers; a four-layer ConvNet with global average pooling encodes exemplars.In zero-shot counting, a learnable [SPE] token replaces the exemplar features.Source: §3.1.1 · pp. 4–5

3.1.2 Feature Interaction Module

• The feature interaction module uses image features as queries and exemplar or [SPE] features as keys and values, combining image self-similarity with exemplar-conditioned comparisons.Its transformer decoder layers preserve the image-feature dimensions throughout interaction.Source: §3.1.2 · p. 5

3.1.3 Decoder

• The decoder reshapes interaction outputs into feature maps, applies four convolution-and-2×-bilinear-upsampling blocks, and regresses a one-channel density heatmap.Source: §3.1.3 · p. 5

3.2 Two-stage Training Scheme

• CounTR first pre-trains its visual encoder with MAE reconstruction from 50% of image patches, then fine-tunes the full counter on density maps using per-pixel mean squared error.Fine-tuning initializes the image encoder from the pre-trained ViT; counts are sums of predicted density values.Source: §3.2 · pp. 5–6

3.3 Scalable Mosaicing

• The mosaic pipeline addresses long-tailed counts by collaging crops from one image to increase object density or four images to diversify backgrounds, then blending image boundaries.Four-image collages support few-shot inference only, whereas same-image collages can train both few-shot and zero-shot settings; blending modifies the image, not its binary dot-annotation map.Source: §3.3 · pp. 6–7

3.4 Test-time Normalisation

• At test time, CounTR divides the density map by the average predicted count at exemplar locations when that average exceeds 1.8, and processes tiny-object images as nine crops when an exemplar side is under 10 pixels.The nine crop predictions are summed; normalization calibrates the expected count of 1.0 at each exemplar position.Source: §3.4 · pp. 7–8, Figure 3 · p. 7

4 Experiments

• Section 4.2 covers model implementation details and the inference-stage design.Source: §4 · p. 8
• Section 4.3 compares CounTR with other counting models, while Section 4.4 tests the two-stage training regime and image mosaicing through ablations.Source: §4 · p. 8
• Sections 4.5 and 4.6 report additional experiments on Val-COCO, Test-COCO, and CARPK, and qualitative visualizations on FSC-147.Source: §4 · p. 8

4.1 Datasets and Metrics

• FSC-147 contains 6,135 images, with object counts ranging from 7 to 3,731 and an average of 56; each image provides three bounding-box exemplars.Source: §4.1 · p. 8
• FSC-147 uses 89 training categories and disjoint validation and test sets of 29 categories each, making it an open-set counting dataset.Source: §4.1 · p. 8
• Model performance is measured with Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE).Source: §4.1 · p. 8
• The metric notation defines NI as the number of test images and Ci and CGT as predicted and ground-truth counts for image i.Source: §4.1 · p. 8

4.2.1 Training Details

• The ViT encoder is pre-trained with MAE [11] on FSC-147 for 300 epochs (batch size 16; learning rate 5×10−6), reconstructing randomly masked 50% of image patches.The 384 × 384 images are split into 16 × 16 patches, and reconstruction uses pixel-wise mean square error.Source: §4.2.1 · pp. 8–9
• Fine-tuning uses AdamW on FSC-147 (batch size 8; learning rate 1×10−5), minimizing scaled mean square error against Gaussian-centered density maps while randomly dropping 20% of pixels.The feature interaction module has two transformer decoder layers; exemplar features are extracted by a four-layer ConvNet, and the image decoder has four up-sampling layers.Source: §4.2.1 · p. 9

4.2.2 Inference Details

• At inference, sliding windows move across images of varying resolutions in 128-pixel strides, and predictions for overlapped regions are averaged.Each window is a fixed-size square matching the size used during training.Source: §4.2.2 · p. 9

4.3 Comparison to state-of-the-art

• CounTR sets new state-of-the-art on FSC-147 for zero-shot and few-shot counting, with especially strong results on the validation set.The comparison is against existing approaches; no table values are provided in the supplied passages.Source: Table 1 · p. 9, §4.3 · p. 9

4.4 Ablation Study

• Image-wise augmentation—including noise, blur, flips, color jitter, and geometric transformations—slightly improves Model B over Model A on validation and test, but its benefit is limited.Source: §4.4 · pp. 9–10
• Self-supervised pre-training significantly improves all validation and test results over directly fine-tuning the ImageNet-pretrained ViT encoder.This comparison is between Model C and Model B.Source: §4.4 · p. 10
• Mosaic further improves performance by adding images with many object instances and distractors from different semantic categories to address long-tailed challenges.Source: §4.4 · p. 10
• Test-time normalisation significantly boosts few-shot counting on both validation and test sets by calibrating predictions with the given exemplar shot.Source: §4.4 · p. 10
• The authors report no tiny difference in final performance across 1, 2, or 3 exemplar shots, describing CounTR as robust under any shot count.Source: §4.4 · p. 10

Val-COCO and Test-COCO

• CounTR halves Mask-R-CNN’s error on Val-COCO and Test-COCO and outperforms FamNet, with 15.16 MAE and 24.29 RMSE on Val-COCO and 11.87 MAE and 14.81 RMSE on Test-COCO.Val-COCO and Test-COCO are FSC-147 subsets collected from COCO.Source: §4.5 · pp. 10–11, Table 3 · p. 11

CARPK

• After fine-tuning on CARPK’s training set and testing with NMS, CounTR performs comparably to existing class-specific car-counting models.CARPK contains 1,448 bird’s-eye images of parking lots.Source: §4.5 · p. 11, Table 4 · p. 11

4.6 Qualitative Results

• CounTR generally counts and locates objects in the first five few-shot FSC-147 examples shown in Figure 4.Figure 4 overlays predicted density maps on the original images.Source: Figure 4 · p. 11, §4.6 · p. 12
• In the last example, bounding-box ambiguity leads CounTR to select spectacle lenses rather than sunglasses, an error corrected by test-time normalisation.Figure 4’s caption defines TTN as test-time augmentation.Source: Figure 4 · p. 11, §4.6 · p. 12
• Figure 5 provides additional qualitative results of CounTR on FSC-147.Source: Figure 5 · p. 12

Conclusion

• CounTR achieves state-of-the-art performance in both zero-shot and few-shot settings.The conclusion reports this result without specifying benchmark values.Source: Conclusion · p. 13
• CounTR is a transformer-based architecture for counting objects from arbitrary semantic categories using arbitrary numbers of exemplars.Source: Conclusion · p. 13
• CounTR is first pre-trained with self-supervised learning and then fine-tuned with supervised learning.Source: Conclusion · p. 13
• The paper proposes a simple, scalable pipeline for synthesizing training images that explicitly forces the model to use given exemplars.Source: Conclusion · p. 13

• The supplementary material introduces the annotation error in 7171.jpg from the FSC-147 test set.Source: A Appendix · p. 16

A.1 On the annotation error of 7171.jpg

• In 7171.jpg, the ground-truth object count conflicts with the exemplars, tending to cause significant evaluation error.Figure 6 shows the ground-truth and exemplar annotations.Source: §A.1 · p. 16, Figure 6 · p. 16