PaperlayerView exact revision on arXiv

Source-linked AI summary

Represent, Compare, and Learn: A Similarity-Aware Framework for Class-Agnostic Counting

Min Shi, Hao Lu, Chen Feng, Chengxin Liu, Zhiguo Cao

arXiv:2203.08354v1cs.CV

TL;DR

Class-agnostic counting aims to count arbitrary-category objects from a few exemplars, but existing methods use predefined similarity rules that can yield unsatisfactory matching. This paper jointly learns representation and similarity in BMNet and BMNet+, reporting state-of-the-art performance on FSC147 and CARPK.

  • Problem

    Existing CAC methods use fixed similarity rules despite differing feature-learning choices, and these rules can yield noisy matches and ambiguous density maps.

    Source: §1 · p. 1

  • Method

    The framework jointly learns representation and similarity end to end, extending bilinear matching in BMNet with self-similarity, dynamic comparison, and explicit similarity-aware supervision in BMNet+.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    BMNet and BMNet+ achieve state-of-the-art performance on FSC147 and CARPK; on FSC147, relative mean absolute error improvements are +33.72% on validation and +33.79% on test.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The paper concludes that similarity modeling matters for CAC and presents joint representation and similarity learning as a generic framework.

    Source: §5 · p. 8

    Better feature representation remains insufficiently addressed; self-similarity may hurt test performance, and some instantiated designs use heuristics.

    Source: §5 · p. 8

Abstract

from arXiv · show

Class-agnostic counting (CAC) aims to count all instances in a query image given few exemplars. A standard pipeline is to extract visual features from exemplars and match them with query images to infer object counts. Two essential components in this pipeline are feature representation and similarity metric. Existing methods either adopt a pretrained network to represent features or learn a new one, while applying a naive similarity metric with fixed inner product. We find this paradigm leads to noisy similarity matching and hence harms counting performance. In this work, we propose a similarity-aware CAC framework that jointly learns representation and similarity metric. We first instantiate our framework with a naive baseline called Bilinear Matching Network (BMNet), whose key component is a learnable bilinear similarity metric. To further embody the core of our framework, we extend BMNet to BMNet+ that models similarity from three aspects: 1) representing the instances via their self-similarity to enhance feature robustness against intra-class variations; 2) comparing the similarity dynamically to focus on the key patterns of each exemplar; 3) learning from a supervision signal to impose explicit constraints on matching results. Extensive experiments on a recent CAC dataset FSC147 show that our models significantly outperform state-of-the-art CAC approaches. In addition, we also validate the cross-dataset generality of BMNet and BMNet+ on a car counting dataset CARPK. Code is at tiny.one/BMNet

1. Introduction

Class-agnostic counting uses a few exemplars to count objects from arbitrary categories, but existing methods’ fixed similarity rules can yield noisy matches. The paper proposes jointly learning representation and similarity, with BMNet and BMNet+ achieving strong benchmark results.

1. Introduction

  • BMNet+ produces higher-fidelity intermediate similarity maps than FamNet, and the paper reports that ablations validate its three main components.The paper also reports cross-dataset generality of its models on CARPK.Source: Figure 1 · p. 1, §1 · p. 2
  • CAC counts objects from arbitrary categories given only a few exemplars, reducing reliance on training data compared with category-specific counting.Category-specific methods typically require numerous training examples.Source: §1 · p. 1
  • Existing CAC methods vary in whether they learn or fix feature extraction, but use predefined similarity rules that can produce noisy matches and ambiguous density maps.FamNet shows background noise and weak target responses in its similarity maps.Source: §1 · pp. 1–2
  • The proposed framework jointly learns feature representation and similarity metric end to end, with BMNet using a learnable bilinear metric instead of a fixed inner product.BMNet+ extends the framework to model similarity through self-similarity, dynamic comparison, and explicit similarity-aware supervision.Source: §1 · p. 2
  • On FSC147, the method reports relative mean absolute error improvements of +33.72% on validation and +33.79% on test.The paper also reports cross-dataset generality on CARPK.Source: §1 · p. 2

2. Related Work

Related work covers category-specific counting, class-agnostic counting, and metric learning. The discussion identifies similarity modeling as an under-addressed issue in CAC and relates it to learned similarity constraints.

2.1. Class-Specific Object Counting

  • Category-specific counting methods include detection, regression, classification, and localization; regression approaches often predict density maps.Other approaches explore architectures, multi-scale strategies, loss functions, reinforcement learning, and localization.Source: §2.1 · p. 2

2.2. Class-Agnostic Counting

  • Earlier CAC methods include matching networks, CFOCNet, and FamNet, but typically emphasize multi-scale strategies, data amplification, or test-time adaptation rather than similarity modeling.FSC147 was proposed as a dataset covering challenges such as occlusion and scale variation.Source: §2.2 · p. 2

2.3. Metric Learning

  • Metric learning pulls similar samples together and pushes dissimilar ones apart; the paper applies this idea to exemplar-target and exemplar-background similarity.The accompanying framework figures contrast prior approaches with joint representation and metric learning in BMNet and BMNet+.Source: §2.3 · p. 2, Figure 3 · p. 3, Figure 2 · p. 3

3. A Similarity-Aware Framework for Class-Agnostic Counting

The framework jointly learns feature representation and similarity metric for class-agnostic counting. It instantiates this approach with BMNet and extends it with dynamic similarity, direct similarity supervision, and self-similarity features.

3.1. Bilinear Matching Network

  • BMNet jointly optimizes representation and similarity using a learnable bilinear metric that captures flexible interactions among feature channels.Its counter combines query features with the similarity map to predict a density map, whose integral gives the count.Source: §3.1 · p. 3

3.2. Learning Dynamic Similarity Metric

  • Exemplar-conditioned channel attention makes the similarity metric dynamic, with more consistent weights for same or visually similar categories than for dissimilar categories.The visualization contrasts apple with apple, strawberry, and stamp.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. Supervising the Similarity Map

  • Direct similarity supervision labels map positions positive when their image blocks contain more than one target and negative when they contain no target.The similarity loss treats exemplar–target matches as signals and exemplar–background matches as noise.Source: §3.3 · p. 4, Figure 5 · p. 4

3.4. Self-Similarity Module

  • Self-attention augments instance features with information from other same-category instances, while scale embeddings compensate for scale information lost during resizing and pooling.The attention maps show query points focusing mainly on same-category instances; scale levels receive learned embeddings added to exemplar features.Source: §3.4 · p. 5

3.5. Implementation Details

  • Experiments reuse FamNet query preprocessing and feature extraction, resize exemplars to 128 × 128, and train end-to-end for 300 epochs.No data augmentation is applied.Source: §3.5 · pp. 5–6

4. Experiments

Experiments compare BMNet and BMNet+ with CAC methods on FSC147, examine BMNet+ components, exemplar counts, and counter inputs, and test generalization to CARPK.

4.1. Comparison With State of the Arts

  • BMNet improves over FamNet by 21.63% validation MAE and 25.93% test MAE; BMNet+ further reduces MAE versus BMNet by 18.23% validation and 12.51% test.Qualitatively, BMNet+ better distinguishes hot-air balloons from a tower, while FamNet fails on strawberries with large intra-class variation and the proposed methods do not.Source: §4.1 · p. 6, Figure 8 · p. 7

4.2. Ablation Study on BMNet+

  • Direct similarity-map supervision yields 3.88% validation and 10.41% test MAE relative improvements; dynamic similarity adds 9.28% validation and 11.29% test MAE relative improvements.Self-similarity with scale embedding improves validation MAE by 0.97, but test performance shows a converse result in B7 versus B9; the authors plausibly attribute this to overfitting from added feature-extractor parameters. Scale embedding generally improves representations.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3. Number of Exemplars per Task

  • More exemplars yield better results, but BMNet with one exemplar surpasses other methods with three; self-similarity offers improvement when exemplars are fewer.FSC147 tests one, two, or three randomly chosen exemplars per task.Source: §4.3 · p. 7, Table 3 · p. 8

4.4. How to Integrate Features for the Counter?

  • Similarity map alone performs worst, while query features help; exemplar-feature concatenation adds marginal gains at extra computational cost, so similarity map plus query features is the default.The authors note that omitting the similarity map makes similarity-metric supervision impossible.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Cross-Dataset Generalization

  • On CARPK, BMNet and BMNet+ gain 49.3% and 63.8% MAE over FamNet, respectively, and retain advantages over FamNet after fine-tuning.The models are pretrained on FSC147; its car category is excluded to avoid category overlap with CARPK, whose images differ substantially from FSC147.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusions and Limitations

The paper concludes that similarity modeling matters for CAC, presenting BMNet and BMNet+ and reporting state-of-the-art performance on FSC147 and CARPK. It also identifies unresolved representation and design limitations.

  • BMNet and BMNet+ achieve state-of-the-art performance on FSC147 and the car-counting dataset CARPK.Source: §5 · p. 8
  • The framework jointly learns feature representation and similarity end to end; BMNet learns bilinear similarity, while BMNet+ adds self-similarity, dynamic similarity, and explicit supervision.Source: §5 · p. 8
  • Feature representation remains under-addressed: self-similarity may hurt test performance, integration of rich representations with similarity maps is unresolved, and some model designs rely on heuristics.Source: §5 · p. 8
Loading 2203.08354v1…