PaperlayerView exact revision on arXiv

Source-linked AI summary

Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challenges

Joana Konadu Owusu, Shivanand Venkanna Sheshappanavar

arXiv:2608.23845v1cs.CV

TL;DR

Object counting has shifted toward open-vocabulary and foundation-model-backed methods, but claims of universal generality exceed the narrow evaluation infrastructure supporting them. The survey addresses this gap with a five-axis taxonomy and cross-domain audit, finding persistent weaknesses in semantic grounding, temporal identity, and occlusion-related spatial reasoning. It concludes that deployment-aware, multimodal evaluation is needed to distinguish open-world generalization from benchmark-specific optimization.

  • Problem

    Claims of general-purpose counting rely on narrow and often saturated benchmarks that do not adequately test semantic grounding, temporal identity, spatial reasoning, or domain transfer.

  • Method

    The survey reviews counting across modalities and domains using a five-axis taxonomy covering modality, mechanism, prompting, supervision level, and generalization setting.

  • Results

    Diagnostic evaluation and cross-domain analysis show that leading methods remain vulnerable to semantic ambiguity, distractors, domain shift, and the tension between general-purpose generality and domain-specific accuracy.

  • Takeaways & Limitations

    Reliable counting requires verifiable instance evidence, distractor rejection, temporal identity preservation, uncertainty exposure, and unified multimodal evaluation.

Abstract

from arXiv · show

Object-counting methods have rapidly shifted from class-specific density regression to open-vocabulary, foundation-model-backed counters. These methods now enumerate instances from various visual and textual prompts. While this shift marks major conceptual progress, our survey argues that claims of universal generality have outpaced the evaluative infrastructure. Most progress metrics rely on a few saturated benchmarks that models exploit for statistical regularities. Newly introduced diagnostic datasets reveal systematic failures in semantic grounding, temporal identity, and spatial reasoning with occlusion. To address these failures, we introduce a five-axis taxonomy (modality, mechanism, prompting, supervision level, and generalization setting). We use this taxonomy to audit the literature across application domains, including microscopy, remote sensing, crowd counting, and agriculture. This formalizes prevailing challenges into six structural contradictions. From these, we propose a roadmap for compositional scene understanding, active counting agents, and unified multimodal evaluation protocols. The main imperative is to build a robust evaluation infrastructure to distinguish open-world generalization from benchmark-specific optimization, rather than simple incremental engineering.

I. INTRODUCTION

Object counting has progressed from class-specific density regression toward cross-modal, open-vocabulary methods, but evaluation remains narrower than claims of generality. The survey unifies this fragmented field with a five-axis taxonomy and audits benchmarks, modalities, domains, and future directions.

  • Methodological progression: Early counting treated dense scenes as class-specific density regression, using point annotations when individual instances were impractical to box or segment.This tradition supported crowd and microscopy counting but did not transfer across object categories.
  • Methodological progression: The survey traces four methodological eras while arguing that benchmark coverage has remained narrow relative to the generality claimed by successive paradigms.The eras mark shifts in approach and are represented through benchmarks and reviewed papers.
  • Evaluation gap: Diagnostic benchmarks expose semantic ambiguity, distractors, prompt sensitivity, and domain-shift vulnerabilities that legacy benchmark leaders still exhibit.The survey contrasts these failures with progress reported on established benchmarks such as FSC-147.
  • Survey framework: The survey reviews more than 600 candidate records and organizes methods through a unified five-axis taxonomy spanning modality, mechanism, prompting, supervision, and generalization.Its contributions also compare specialized-domain transfer and formalize gaps into six structural contradictions with three future research directions.
  • Cross-modal scope: Object counting has expanded across image, video, 3D, multi-view, and specialized sensing, with each modality changing the evidence available to the model.Video addresses temporal identity, depth-based modalities address occlusion ordering, and thermal sensing addresses poor visibility.

A. Density and Crowd Counting

Counting surveys and methods reveal a field shaped by density, occlusion, domain constraints, and increasingly broad target specifications. The survey connects these fragmented literatures through a cross-modal taxonomy while emphasizing that scalar count accuracy alone cannot establish semantic or instance-level validity.

  • Density and crowd counting: Crowd-counting reviews emphasize density regression, scale variation, occlusion, and count-error metrics, while treating density estimation as the primary mechanism.Weak spatial cues remain important when individual object separation fails.
  • Class-agnostic counting: Open-world text-guided counting captures class-agnostic and zero-shot formulations but excludes video, multimodal reasoning, and 3D modalities.This leaves broader cross-modal comparisons outside its scope.
  • Survey coverage: Existing reviews are fragmented across open-world counting, crowd density, microscopy, agriculture, and multimodal reasoning, with each emphasizing a narrower domain or task setting.Application-specific reviews preserve constraints such as growth stages, water turbidity, and expert-defined biological boundaries.
  • Unified framework: The survey bridges these literatures by unifying density regression, geometry-aware methods, tracking, and foundation models under five structural axes rather than application domains alone.The taxonomy compares what methods observe, how they convert evidence, specify targets, receive supervision, and generalize.
  • Counting as visual reasoning: A scene can yield different valid counts for prompts such as “vehicles”, “parked cars”, or “moving cars”, so the count must remain tied to target definition and visual evidence.Providing the right number for the wrong query is characterized as a reasoning failure rather than a counting error.
  • Mechanism trade-offs: Counting mechanisms differ in their evidence trade-offs: density maps handle dense regimes, detection supports auditing, and segmentation provides stronger instance support but depends on mask quality.The survey notes that count error alone is weak evidence for class-agnostic generalization.

B. Formal Problem Definition & Algorithmic Paradigms

Object counting estimates the number of query-conforming instances from visual or multimodal evidence. Its four paradigms differ in how the query interacts with evidence, ranging from density integration to spatially grounded or language-mediated outputs.

  • The task estimates the number of instances matching query q in an image, video, point cloud, or multimodal input X.
  • The four paradigms progress from class-specific density regression to exemplar-based cross-attention, promptable localization, and multimodal language reasoning.
  • Density regression treats q as implicit and integrates a predicted spatial density map, whereas exemplar counting uses visual crops as attention keys and values.
  • Promptable concept localization grounds multimodal prompts into instance masks or bounding boxes before counting detected objects.
  • Multimodal reasoning can combine visual, audio, and textual representations to generate a count directly or through sequential, inspectable spatial judgments.
  • The metric–paradigm matrix highlights that evaluation metrics are not uniformly computable across counting mechanisms.

C. Evaluation Objectives and Metric Incompatibilities

Counting evaluation must distinguish scalar count accuracy from semantic and spatial correctness. The supplied passages show that standard metrics can conceal wrong-instance predictions and are incompatible with some mechanism-specific outputs.

  • MAE and RMSE are dominant scalar metrics computed between predicted and ground-truth counts over a test set.
  • Localization metrics such as Average Precision at an IoU threshold cannot be computed for density-based methods.
  • Tracking-based unique-instance counting requires identity-sensitive metrics because scalar counts cannot distinguish re-entry overcounting from missed-detection undercounting.
  • The semantic grounding illusion exposes why zero scalar error does not establish that the correct category or instances were counted.
  • A model can achieve MAE = 0 while compensating false positives and false negatives, including counting distractors instead of the queried instances.

IV. TAXONOMY OF OBJECT COUNTING METHODS

The taxonomy organizes counting methods by five structural axes that describe evidence, mechanism, query specification, supervision, and generalization scope. It separates these design properties from evaluation and deployment metrics to clarify what methods can handle in new settings.

  • The five axes are modality, counting mechanism, prompting technique, supervision level, and generalization setting.
  • A counting claim combines evidence X, query q, mechanism m, and supervision S, while generalization G and evaluation protocol E remain external interpretive layers.
  • Modality: Modality determines available evidence across image, video, temporal, thermal, 3D, and point-cloud settings.
  • Counting mechanism: Mechanisms include density regression, detection, segmentation, tracking, and direct count reasoning, with different spatial accountability and occlusion sensitivities.
  • Prompting: Prompting specifies targets through fixed labels, exemplars, text, points, boxes, or complex multimodal instructions, shaping interface flexibility.
  • Supervision level: Supervision ranges from spatial instance labels to aggregate, mixed, self-supervised, annotation-free, or no counting-specific training, trading spatial accountability against annotation cost.
  • Generalization setting: Prompting and generalization must be distinguished because their conflation obscures benchmark meaning and enables semantic grounding failures.

A. Density-Map and Regression-Based Counting

Counting methods progress from density-map regression toward exemplar-, text-, detection-, and segmentation-conditioned mechanisms, while different outputs expose different evidence and failure modes.

  • Density-map regression: Density-map regression converts point annotations into Gaussian-smoothed maps, predicts them from images, and integrates the map to obtain the count.This canonical workflow supports dense, class-specific scenes but produces an aggregate spatial estimate rather than discrete instances.
  • Density-map regression: Fixed kernels create a scale problem and domain-locking because kernel radius, density range, and visual statistics are fixed during training.Later methods address scale variation, but the cited discussion identifies domain-locking as the deeper unresolved limitation.
  • Prompt-conditioned counting: Exemplar- and text-conditioned similarity mechanisms replace fixed class labels with cross-attention or similarity signals, enabling conditioning on visual examples or text.The literature includes learned cross-attention, metric learning, prototype adaptation, and zero-shot text-guided density estimation.
  • Detection-based counting: Detection-based counting enumerates localized points, boxes, maxima, or proposals, making predicted evidence discrete and directly auditable.P2PNet illustrates direct point-proposal regression, while density-normalized Average Precision supplements count-only evaluation.
  • Segmentation-based counting: Segmentation-based counting estimates masks or regions whose quality directly controls count quality when adjacent instances merge or objects are over-segmented.Masks provide stronger instance-level evidence, but the mechanism is reliable only when segments correspond to countable objects.

D. Discussion

Video counting separates frame-visible estimation, persistent-identity counting, and event or spatial-aggregation counting, but evaluation often conflates these targets and their distinct failure modes.

  • Frame-visible counting: Frame-level video methods estimate visible occupancy per frame or short temporal window, using temporal information mainly to smooth or stabilize predictions.These methods do not generally determine whether an object reappearing after occlusion is the same identity.
  • Identity counting: Tracking-based methods target unique-instance counts through detection, association, re-identification, and duplicate suppression across time.Their central decision is whether repeated visual evidence belongs to an existing identity or a new object.
  • Identity counting: Tracking errors become counting errors, so scalar accuracy can conceal a missed object offset by a double count and requires identity-switch metrics.This creates a video-level mismatch between numerical counts and identity-aware evidence.
  • Event and spatial aggregation: Trajectory-event and spatial-aggregation methods count crossings, periodic actions, or reconstructed regions rather than only physical objects visible in individual frames.Examples include adaptive counting lines, repetition estimation, and mosaicking overlapping frames into a larger scene.
  • Event and spatial aggregation: Camera placement, line definition, stitching errors, and variable repetition tempo can determine performance in event and spatial-aggregation counting.These factors define the operational task rather than functioning as minor implementation details.

D. Discussion

Geometric modalities change the physical evidence available for counting by addressing depth-axis occlusion, while evaluation must distinguish depth augmentation, true 3D structure, and multi-view evidence.

  • Geometric counting: Two-dimensional methods assume occlusion occurs when object projections overlap, but stacked or foreshortened instances can violate that image-plane assumption.Depth-axis arrangements expose hidden structure that RGB projections collapse.
  • Depth-aware counting: Depth-aware counting adds distance and surface-discontinuity cues to 2D imagery, making some visually similar or partially visible instances more separable.These methods remain depth-augmented 2.5D systems because their density maps operate on a 2D image grid.
  • True 3D counting: True 3D counting uses point clouds, voxels, or reconstructed meshes to reason about spatial occupancy, hidden surfaces, and stacked objects.Its evidence is geometric rather than limited to a projected image plane.

C. Multi-View Counting

Multi-view and foundation-model approaches broaden counting evidence and semantic interfaces, but deployment still depends on reliable instance separation and domain-specific evaluation.

  • C. Multi-View Counting: Multi-view methods fuse viewpoints using calibration, vanishing points, or view correspondence to reveal occluded objects and reduce single-view uncertainty.They occupy an intermediate position between purely 2D and fully reconstructed 3D counting.
  • C. Multi-View Counting: Cross-view fusion can double-count overlapping observations, while calibration errors, asynchronous capture, and partial overlap corrupt counts.These risks require distinguishing multi-view feature fusion from 3D reconstruction.
  • B. Discussion: Foundation-model counting produces generated language outputs from autoregressive multimodal models instead of regression heads, density maps, or bounding-box counts.The family ranges from untouched zero-shot inference to counting-specific post-training and from text-only to cross-modal reasoning.
  • B. Discussion: Counting-specific effort in foundation-model systems is applied through inference pipelines, model adaptation, or training curricula while the general-purpose model remains the semantic and reasoning engine.Examples include divide-and-conquer inference, weak supervision, synthetic spatial examples, and multimodal adaptation.
  • D. Discussion: Application-specific systems must evaluate whether measurements, crop estimates, or inspection counts are reliable enough for professional workflows beyond benchmark accuracy.Operational constraints therefore become part of the counting problem rather than an afterthought.

A. Monitoring, Agricultural, and Marine Systems

Application-specific counting spans agriculture, microscopy, remote sensing, marine monitoring, and industrial systems, where sensing and deployment constraints shape both methods and evaluation. Benchmarks and protocols remain fragmented, limiting cross-domain assessment of generality.

  • Monitoring and agriculture: Agricultural and ecological systems use counting for monitoring, resource management, and population or crop-related measurement across specialized sensing settings.Examples include wheat spike counting and neuromorphic event-stream insect counting.
  • Scientific and medical systems: Microscopy counting targets cells, bacteria, and instruments defined by expert protocols, using segmentation or density regression to handle clumping and overlap.These systems address fluorescent and standard microscopy scenes where individual separation is difficult.
  • Remote sensing: Remote-sensing counters must handle small, dense, variably scaled, cluttered, and arbitrarily oriented targets, while prior methods commonly assume fixed category vocabularies.Counting genuinely novel classes therefore requires costly re-annotation in the described setting.
  • Industrial and deployment constraints: Industrial and deployed counters prioritize repeatability, throughput, occlusion robustness, association, runtime, memory, power, privacy, sensor physics, and spatial verifiability.These requirements can dominate feasibility in production, edge, video, transportation, and scientific workflows.
  • Benchmark infrastructure: 0?

C. Video and Temporal Benchmarks

Video, depth, multimodal, and application-specific benchmarks extend counting beyond static RGB, but they use heterogeneous datasets and metrics for temporal identity, spatial reasoning, occlusion, and semantic intent. This landscape makes cross-setting comparison difficult.

  • Video benchmarks: Video-counting datasets remain scarce, with benchmarks combining detection, tracking, and counting or targeting identity-consistent pedestrian tracking.The cited video datasets address general-purpose counting, dense drone crowds, and identity-preserving tracks.
  • Depth and non-RGB benchmarks: Depth-aware and RGB-D datasets are introduced for specific sensing needs, including crowd data, depth-aware zero-shot counting, and indiscernible marine-object counting.These datasets are generally tied to the modalities or methods they support.
  • Multimodal reasoning: VLM benchmarks test relational questions, occlusion, prompt intent, spatial composition, and amodal counting beyond simple object detection.Diagnostic studies also connect errors to patchification, text priors, and implicit counts.
  • Application-specific benchmarks: Application datasets cover remote sensing, marine and wildlife monitoring, agriculture, transport, and surveillance with domain-specific categories and conditions.Examples include RGB/NIR imagery, sonar-video tracking, plant taxonomy, produce, microalgae, and overlapping vehicles.
  • Annotations and metrics: Counting evaluation uses different supervision forms and metrics, including points, boxes, masks, MAE, RMSE, Exact-Match, Threshold Accuracy, and GAME.GAME adds spatial subdivision, while tracking and open-vocabulary settings require outputs beyond a scalar count.

I. Reproducibility, Protocol Alignment, Efficiency Reporting

Counting evaluation remains difficult to reproduce and align because protocols, computational reporting, and metrics differ across paradigms. The survey argues that evaluation should match each method’s claimed capability rather than rely on scalar benchmark error alone.

  • Reproducibility: Cross-paper comparisons are hindered by inconsistent data splits, prompt selection, seed reporting, and scarce computational-efficiency reporting.This especially limits comparisons between lightweight models and large foundation models in constrained deployments.
  • Protocol alignment: Evaluation should pair counting claims with capability-aligned evidence, including localization or masks, identity switches, 3D completeness, distractor rejection, and prompt robustness.The required evidence differs across 2D, video, 3D, and open-vocabulary settings.
  • Generalization and localization: No single surveyed method combines strong zero-shot generalization with precise spatial localization.Vision-language counters use coarse patch-level attention, whereas density architectures provide finer localization but depend on class-specific priors.
  • Density range: No surveyed method performs competitively across the full density continuum from fewer than five instances to over a thousand.Density models can phantom-count sparse scenes, while exemplar matching can undercount heavily overlapping objects.
  • Temporal identity: Tracking systems face a temporal identity trade-off: closed-set trackers fail on unseen classes, while identity-free aggregators double-count re-entering objects.Motion and geometry alone remain insufficient for re-identification in the surveyed setting.
  • Benchmark validity: Improving FSC-147 MAE can create a false impression of progress because diagnostic benchmarks expose semantic, compositional, and adversarial weaknesses.MAE alone cannot identify whether spatial detection, semantic classification, or cardinality aggregation is the bottleneck.

F. Fragmented Modalities and Deployment Efficiency

The survey identifies fragmented modalities, deployment constraints, and benchmark saturation as barriers to reliable general-purpose counting. It proposes structured scene representations, active information-seeking agents, unified multimodal architectures, and stronger evaluation practice.

  • Fragmented modalities and deployment: Non-RGB modalities provide physical evidence unavailable to RGB, but private incompatible benchmarks prevent cross-modal comparison and accuracy-only evaluation obscures deployment feasibility.Camera placement and edge-compute constraints matter alongside architectural design.
  • Benchmark saturation: Incremental backbone scaling and exemplar retrieval can raise saturated benchmark scores while leaving structural failure modes unresolved.The survey characterizes these directions as less transformative than approaches targeting the identified contradictions.
  • Compositional scene understanding: Future counters should build compositional scene representations linking regions through proximity, occlusion, and part-whole relations so counts become verifiable structural properties.Segmentation, geometric priors, point-prompted counting, and sequential spatial reasoning provide early examples.
  • Interactive agents: Active counting agents could request targeted information such as zooms, human clarification, or adjacent video frames when uncertainty, prompt brittleness, or re-entry events require it.Interactive corrective feedback and cross-modal clue grounding motivate explicit belief-map maintenance.
  • Unified multimodal evaluation: A shared encoder with lightweight modality-specific embeddings and a common counting decoder could evaluate RGB, video, depth, thermal, and point-cloud inputs without modality-specific fine-tuning.Hyperspectral counting lacks a dedicated benchmark, while event-camera counting remains at an early-demonstration stage.
  • Conclusion: The survey’s central finding is that claims of generality outpace verification, while domain-specific systems still outperform general-purpose foundation models under binding physical constraints.No method currently resolves the tension between cross-domain generality and domain-specific accuracy.
  • Conclusion: Reliable general-purpose counting requires identifying instances, rejecting distractors, preserving temporal identity, and exposing uncertainty across diverse real-world settings.The survey frames these requirements as boundaries on claims of universal counting.
Loading 2608.23845v1…