PaperlayerView exact revision on arXiv

Source-linked AI summary

RECOUNT: Reference-guided Counting with Synthetic Visual Exemplars

Adriano D'Alessandro, Ali Mahdavi-Amiri, Ghassan Hamarneh

arXiv:2608.20621v1cs.CV

TL;DR

Existing zero-shot counters localize objects but poorly distinguish fine-grained categories, while few-shot counters require annotations on every query image. RECOUNT uses one off-scene reference plus diffusion-generated target and distractor exemplars to add visual categorization to a frozen counter. It achieves state-of-the-art zero-shot counting across benchmarks, reducing error by 55% on LookAlikes and 21% on PairTally relative to the strongest prior zero-shot counter.

  • Problem

    Text prompts underspecify fine-grained visual identity, while few-shot visual exemplars require manual query-image annotations.

  • Method

    RECOUNT attaches a lightweight categorization module to a frozen counter and matches localized proposals against diffusion-generated target and distractor galleries trained with SYNTHALIKES.

  • Results

    RECOUNT sets a new zero-shot state of the art on LOOKALIKES, PairTally, and PrACo without query-image annotation or counter retraining.

  • Takeaways & Limitations

    Image-guided zero-shot counting combines reusable visual category specification with the annotation-free operation of a text prompt.

  • Takeaways & Limitations

    Dense regions can place multiple instances in one backbone feature cell, blending descriptors and degrading candidates packed with distractors.

Abstract

from arXiv · show

Text-guided zero-shot object counters excel at spatial localization but categorize poorly on novel or fine-grained classes: natural language is too coarse to fully specify visual identity, so they fail to separate visually similar distractors. Few-shot counters sidestep this with visual exemplars, but require manual annotations on every image. To resolve this dilemma, we introduce RECOUNT, a plug-and-play framework for image-guided zero-shot counting. Rather than specify a category with a text prompt, our key insight is to specify it visually, from a single off-scene reference image. However, we find that a lone reference image provides narrow coverage of a category's appearance and is unreliable across diverse scenes. We therefore repurpose a diffusion model as an automated contrastive data engine that expands the reference into a diverse exemplar gallery, supplying the discriminative detail that text cannot. RECOUNT preserves the class-agnostic proposals of any frozen counter and offloads categorization to a separate visual module (a frozen backbone with a lightweight head trained on this synthetic data) that matches each proposal against the target and distractor galleries. Applied to a frozen counter, RECOUNT attains the best zero-shot accuracy on both benchmarks, cutting counting error (MAE) by 55% on LookAlikes and 21% on PairTally relative to the strongest prior zero-shot counter.

1 Introduction

RECOUNT addresses the gap between text-guided zero-shot counting and annotation-heavy few-shot counting by specifying categories visually and expanding one reference into synthetic exemplars. It preserves frozen counters’ localization while adding visual discrimination, achieving state-of-the-art zero-shot results without query-image annotations.

  • Motivation: Text-guided counters localize objects effectively but struggle to distinguish novel categories, visually similar distractors, and background clutter.Text prompts cannot fully express fine-grained visual attributes, while counting datasets provide abundant localization annotations but fewer than 100 training categories.
  • Image-guided zero-shot counting: RECOUNT specifies a target category with one off-scene reference image reused across query images, avoiding manual annotations on each query.This image-guided zero-shot paradigm aims to combine few-shot visual precision with text-prompt scalability.
  • Synthetic exemplars: A diffusion model expands each reference into diverse synthetic visual exemplar galleries for the target and its distractors.The generated exemplars vary pose, lighting, and arrangement while deriving object identity from the reference image.
  • RECOUNT: RECOUNT attaches to a frozen counter, leaving localization unchanged while a lightweight visual module performs contrastive categorization.The module pools frozen-backbone features at candidate points, projects them with a lightweight head, and trains on synthetic data.
  • Results: RECOUNT cuts counting error by 55% on LookAlikes and 21% on PairTally relative to the strongest prior zero-shot counter.The framework establishes state-of-the-art zero-shot performance on both benchmarks without annotating query images.

2 Related Work

Prior work establishes a tension between visual precision and scalable category specification: few-shot methods use annotated exemplars, whereas zero-shot methods use text and often rely on imperfect pseudo-exemplars. RECOUNT instead decouples localization from categorization and grounds synthetic, contrastive training in off-scene visual references.

  • Few-Shot Counting: Few-shot counters use visual exemplars that match query-image distributions precisely but require manual annotations on every query image.Their visual specification captures object identity more precisely than language prompts.
  • Zero-Shot Counting: Zero-shot counters specify categories with text prompts and increasingly predict explicit point or box locations.This line of work includes methods built on grounding models such as GroundingDINO.
  • Pseudo-exemplars: Text-guided pseudo-exemplar methods select query-image features, but mixed-category scenes can cause text to select incorrect target features.Their precision remains dependent on whether the prompt identifies the right visual features.
  • Synthetic Data: Synthetic-data approaches support discrimination through rendered or diffusion-generated data, including test-time adaptation methods for adding categories.These approaches differ in whether generation is grounded in text, query images, or other synthetic sources.
  • RECOUNT’s distinction: RECOUNT grounds generation in one off-scene reference and uses one engine both to train its discrimination head and specify targets online.The frozen base counter avoids the per-category retraining used by test-time adaptation.

3 RECOUNT

RECOUNT decomposes counting into class-agnostic localization and dense categorization, then leaves the former to a frozen counter. Its generative data engine addresses both category specification at inference and category recognition during training.

  • Problem decomposition: Counting arbitrary categories combines dense localization with dense categorization: finding objects and deciding which belong to the target.Categorization is identified as the bottleneck despite data-rich localization.
  • Design principle: RECOUNT leaves localization to a frozen counter and focuses on recognizing target objects while rejecting visually similar categories.This separation treats counting as the composition of localization and categorization.
  • Generative data engine: The same generative data engine specifies targets at inference and supplies missing training signal for dense categorization.Online it creates visual support sets; offline it produces synthetic dense scenes spanning more than a thousand categories.

3.1 A Synthetic Data Engine for Dense Categorization

The synthetic data engine uses reference-guided diffusion and fine-grained taxonomies to create dense contrastive training scenes and reusable target-support galleries. SYNTHALIKES supplies hard-negative supervision, while synthetic visual exemplars specify categories without query-image annotation.

  • Reference-Guided Image Generation: Reference-guided diffusion preserves fine-grained object identity that text-only generation may replace with generic category traits.Prompt templates then place the referenced identity into varied scenes, viewpoints, arrangements, and lighting.
  • Automated Data Engine: The data engine combines taxonomies, retrieved reference images, quality filtering, LLM scene templates, and latent diffusion to synthesize dense categorized scenes.Its taxonomy merges iNaturalist with manually curated non-natural classes and spans over a thousand categories.
  • The SYNTHALIKES Dataset: SYNTHALIKES trains dense categorization by pairing each category with visually similar distractors that must be rejected.Taxonomic siblings serve as hard negatives because they are especially easy to confuse.
  • The SYNTHALIKES Dataset: Generated scenes are organized into contrastive sibling groups such as flower species, shellfish, and butterflies.These multi-object groups provide explicit training signal for the downstream categorization module.
  • Dataset Construction: Synthetic scenes are pseudo-labeled with a frozen class-agnostic counter, requiring approximate marking of sufficient category instances rather than perfect localization.Each image receives a fine-grained category name and point annotations for target instances.
  • Synthetic Visual Exemplars: At inference, one reference image produces positive target and negative distractor galleries of synthetic visual exemplars.The galleries cover varied appearances and are built once per category, reused across images, and require no target-image annotation.

3.2 Surfacing Discriminative Visual Features

RECOUNT learns a category-agnostic projection head that exposes fine-grained visual differences by contrasting target exemplars against synthetic distractors, while keeping the backbone and counter frozen.

  • Motivation: Raw backbone similarity collapses visually similar distractors because broad category signals dominate the subtle cues needed for separation.RECOUNT addresses this by projecting pooled candidate and exemplar features into a refined embedding space.
  • Contrastive Training: Synthetic scenes provide per-object training embeddings by combining frozen-counter point proposals, pooled backbone features, and projection through Pθ.The training pipeline uses synthetic scenes and pseudo-labels rather than manually annotated query images.
  • Contrastive Training: Multi-negative InfoNCE pulls each object embedding toward its category’s positive gallery and away from sibling and cross-group distractors.The negatives target precisely the categories whose features are ambiguous in the backbone representation.
  • Point Pooling: Each synthetic visual exemplar is reduced to one prototype by point-indexing backbone features, averaging them, and projecting the resulting descriptor.A support gallery contains one prototype per exemplar.
  • Contrastive Training: Only Pθ is trained; the backbone and counter remain frozen, while a dispersion regularizer prevents positive prototypes from collapsing together.The residual projection head preserves the backbone’s geometry while surfacing discriminative features.

3.3 Inference

At inference, RECOUNT classifies frozen-counter proposals by comparing projected candidate descriptors with positive target and negative distractor galleries. Crop-and-reprocess supplies finer features for dense regions where the base grid merges instances.

  • Gallery Matching: Each candidate is projected and compared with positive target prototypes and negative distractor prototypes from the support set.The support galleries contain one prototype per exemplar.
  • Gallery Matching: A candidate is accepted when it more closely resembles the target than any distractor, and the predicted count is the number accepted.The nearest-exemplar rule uses the margin m(p) = s+(p) − s−(p), accepting candidates when m(p) > 0 without a per-category threshold.
  • Dense Regions: Fixed feature grids can merge several dense instances into one descriptor, causing distractors among targets to inherit target-like appearance.This degrades categorization especially for the most crowded candidates.
  • Dense Regions: Crop-and-reprocess detects dense clusters, re-extracts native-resolution features within padded crops, and re-categorizes the enclosed candidates.For dense candidates, RECOUNT keeps the decision from the base or crop descriptor with the larger absolute margin.

4 Experiments

RECOUNT improves zero-shot counting by separating dense categorization from localization and using reference-guided synthetic exemplars. Experiments show gains across datasets, backbones, reference settings, gallery sizes, and dense spatial conditions.

  • Benchmark results: 20.59% improvement reduces PairTally test-set MAE from 50.32 to 39.96 over zero-shot CountGD.On LOOKALIKES, RECOUNT improves MAE by 72.12% over CountGD and reduces it by 54.6% over GroundingREC.
  • Benchmark results: RECOUNT outperforms CountGD++ in zero-shot evaluations without additional training, indicating stronger categorization despite similar localization capabilities.The comparison covers LOOKALIKES and PairTally using positive and negative text and pseudo-exemplar settings.
  • Prompt adherence: RECOUNT achieves an NMN of 0.02 and PCCN of 98.15 on the Negative Test, while matching CountGD++ on the Mosaic Test.These results assess suppression of incorrect-category counts and positive-class counting performance.
  • Backbone and gallery ablations: Across DINOv3, DINOv2, and FLAIR, SVE galleries outperform single-reference matching, and the RECOUNT head adds consistent gains.On DINOv3, RECOUNT adds 33.5 recall points and 12.5 AUC points over dino.txt despite using far less training data.
  • Synthetic data: Scaling from 1 to 12 SVEs yields a 5.2-point recall gain before performance largely saturates.This ablation measures categorization directly at ground-truth object locations.
  • Synthetic data: Localized pooling at pseudo-detection coordinates improves performance over global average pooling, which suffers from background contamination.The comparison evaluates how SVE feature grids are converted into category representations.
  • Synthetic data: Reference-guided generation improves recall by 6.3 points and AUC by 2.6 points over text-only generation, while removing references causes a 14.3-point recall drop.The ablation evaluates reference use during training and inference on LOOKALIKES with oracle localizations.
  • Spatial dynamics: Crop-and-reprocess improves contaminated recall by 13.3 points for targets near distractors in dense regions.Spatial analysis attributes degradation to feature mixing and receptive-field leakage from neighboring distractors.

5 Conclusions

RECOUNT decouples localization from categorization and specifies categories with images rather than words, using synthetic visual exemplars to enable precise counting in cluttered scenes. Applied to frozen counters, it achieves state-of-the-art results without query-image annotation or counter retraining.

  • RECOUNT decouples dense localization from dense categorization by specifying categories with images instead of words.
  • A single real reference image generates synthetic visual exemplar galleries and the SYNTHALIKES corpus, converting open-world categorization into exemplar matching.
  • RECOUNT sets a new state of the art on LOOKALIKES, PairTally, and PrACo without query-image annotation or counter retraining.
  • Training the lightweight discrimination head on SYNTHALIKES improves categorization across several frozen backbones, indicating the gains are not tied to one representation.
  • RECOUNT offers a scalable path to precise counting in cluttered, look-alike-rich scenes where language alone falls short.
Loading 2608.20621v1…