PaperlayerView exact revision on arXiv

Source-linked AI summary

BSNet: Bi-Similarity Network for Few-shot Fine-grained Image Classification

Xiaoxu Li, Jijie Wu, Zhuo Sun, Zhanyu Ma, Jie Cao, Jing-Hao Xue

arXiv:2011.14311v1cs.CV

TL;DR

Few-shot fine-grained classification requires discriminative features from few labeled examples, while metric-based methods often rely on one similarity measure. BSNet shares an embedding module across two similarity branches, and experiments report improved or matched state-of-the-art performance across fine-grained benchmarks.

  • Problem

    Few-shot fine-grained classification must learn discriminative features from few labeled images, but most metric-based methods use a single similarity measure and feature space.

  • Method

    BSNet uses a shared convolution-based embedding module and a bi-similarity module with two similarity measures and jointly trained heads.

  • Results

    BSNet improves or matches previous state-of-the-art performance on fine-grained image datasets and often outperforms individual similarity networks.

  • Takeaways & Limitations

    BSNet learns fewer but more discriminative regions and can reduce model complexity relative to single-similarity networks despite having more parameters.

Abstract

from arXiv · show

Few-shot learning for fine-grained image classification has gained recent attention in computer vision. Among the approaches for few-shot learning, due to the simplicity and effectiveness, metric-based methods are favorably state-of-the-art on many tasks. Most of the metric-based methods assume a single similarity measure and thus obtain a single feature space. However, if samples can simultaneously be well classified via two distinct similarity measures, the samples within a class can distribute more compactly in a smaller feature space, producing more discriminative feature maps. Motivated by this, we propose a so-called \textit{Bi-Similarity Network} (\textit{BSNet}) that consists of a single embedding module and a bi-similarity module of two similarity measures. After the support images and the query images pass through the convolution-based embedding module, the bi-similarity module learns feature maps according to two similarity measures of diverse characteristics. In this way, the model is enabled to learn more discriminative and less similarity-biased features from few shots of fine-grained images, such that the model generalization ability can be significantly improved. Through extensive experiments by slightly modifying established metric/similarity based networks, we show that the proposed approach produces a substantial improvement on several fine-grained image benchmark datasets. Codes are available at: https://github.com/spraise/BSNet

I. INTRODUCTION

Few-shot fine-grained classification must learn discriminative features from few labeled examples, while metric-based methods commonly rely on a single similarity measure. BSNet addresses this similarity bias by combining two measures in one network.

  • Few-shot learning aims to learn latent patterns from few labeled images because deep models typically require many labeled instances.
  • Fine-grained datasets contain many similar subcategories with limited data, making discriminative feature learning from few labels especially challenging.
  • Metric-based few-shot methods generally use a single similarity measure, which may produce similarity-biased features and lower generalization when training data are scarce.
  • BSNet combines two similarity measures with a shared embedding module and improves four state-of-the-art methods on four fine-grained benchmark datasets.
  • BSNet contains more parameters yet has lower model complexity than the mean complexity of two single-similarity networks, and visualization shows discriminative input regions.
  • The proposed approach introduces bi-similarity into metric-based few-shot learning rather than learning a prototype.

III. THE PROPOSED BI-SIMILARITY NETWORK

Few-shot classification is formulated as C-way K-shot prediction over novel classes, using episodic tasks with support and query sets. BSNet is evaluated through meta-training, validation, and testing on disjoint label sets.

  • In C-way K-shot classification, models receive K labeled images from each of C classes and classify unlabeled images from novel classes after training.
  • The dataset is divided into training, validation, and test parts with disjoint label sets.
  • BSNet uses one embedding module followed by a bi-similarity module that outputs two similarity scores for each query and class prototype.
  • Each episodic training task randomly selects C classes and M images per class, then separates them into K-shot support and M −K query images.
  • Meta-training learns transferable bi-similarity knowledge, validation tunes hyperparameters, and testing reports mean meta-testing accuracy.

B. Bi-Similarity Network

The BSNet bi-similarity network shares one embedding module and uses two measurement branches to compare query representations with class representations. Its outputs provide two class-level similarity predictions.

  • BSNet contains one embedding module fφ followed by two similarity measurement branches, gϕ and hγ.
  • The embedding module processes support and query images before their representations enter the bi-similarity module.
  • For each query image and class, the bi-similarity module generates two similarity scores and corresponding one-hot class predictions.
  • Each prediction selects the class with the maximum score from one of the two similarity branches.
  • Training compares both branch predictions with the ground-truth one-hot label and averages their losses for back-propagation.
  • The number of query images, |Qi|, determines the aggregation of task-level losses.

3) The Validation/Testing Procedure:

During validation and testing, BSNet produces two class assignments for each query and selects the class associated with the maximum average similarity score. The procedure then evaluates mean accuracy across episodes.

  • 3) The Validation/Testing Procedure:: Validation and testing assign each query image to a class using the maximum average similarity score.
  • 3) The Validation/Testing Procedure:: During episodic training, tasks are generated from Dtrain and split into support and query sets before similarity scores are computed.
  • 3) The Validation/Testing Procedure:: The algorithm generates two class assignments by taking arg max over the S1 and S2 similarity scores.
  • 3) The Validation/Testing Procedure:: Each assignment is converted into a corresponding one-hot prediction vector according to Equation (1).
  • 3) The Validation/Testing Procedure:: The procedure updates model parameters with the task loss using an optimizer and reports mean accuracy after training stops.

C. The Design of BSNet

BSNet combines a convolution-based feature embedding module with two complementary similarity modules. The design supports prototype-based K-shot processing and can pair a self-designed cosine module with established similarity modules.

  • The convolution-based embedding module generates representations for query and support images before bi-similarity processing.
  • The bi-similarity module produces two similarity measurements between each query image and class-conditioned support representations.
  • The relation module processes concatenated query and support representations through convolution blocks and fully connected layers to produce C similarity scores.
  • For K-shot classification, BSNet averages each class’s support feature maps into C prototypes before concatenating them with the query feature map.
  • BSNet combines a self-designed cosine similarity module with similarity modules from Matching Network, Prototype Network, and DN4.
  • After computing two similarity scores for each query-class pair, the model uses separate strategies for training and validation/testing.

D. The Empirical Rademacher Complexity of BSNet

The analysis defines BSNet as sharing one feature embedding between two similarity networks while retaining their separate similarity modules. Its empirical Rademacher complexity is bounded by the average complexity of the individual networks.

  • BSNet shares a common feature embedding between two similarity networks while adopting their separate similarity modules.
  • The Rademacher complexity measures the richness of a function family and is defined empirically with respect to a fixed sample and random sign variables.
  • Theorem 1 bounds BSNet’s empirical Rademacher complexity using the complexities of the two single-similarity network families.
  • Despite having more model parameters, BSNet’s Rademacher complexity is no more than the average of the complexities of two individual networks.

IV. EXPERIMENTAL RESULTS AND DISCUSSIONS

The experiments assess BSNet against state-of-the-art methods, across backbone changes, individual similarity modules, reduced feature counts, and alternative similarity-module choices.

  • The evaluation compares BSNet with state-of-the-art methods on few-shot fine-grained classification.
  • The experiments investigate BSNet’s generalization ability by changing the backbone network.
  • Ablation analyses study the effectiveness of each similarity module in BSNet.
  • The evaluation examines whether BSNet can learn a reduced number of class-discriminative features.
  • The experiments investigate the effect of changing the similarity modules used in BSNet.

A. Datasets

Experiments use four fine-grained image benchmarks with meta-training, validation, and test splits. The datasets cover aircraft, cars, dogs, and CUB-200-2011, with category-rich fine-grained structure.

  • Experiments use FGVC-Aircraft, Stanford-Cars, Stanford-Dogs, and CUB-200-2011 as four benchmark fine-grained datasets.Each dataset is divided into meta-training, meta-validation, and meta-test sets in a 2 : 1 : 1 ratio; images are resized to 84 × 84 without boundary boxes.
  • FGVC-Aircraft contains hierarchical annotations and is evaluated at the variant level using 100 categories.
  • Stanford-Cars contains 16,185 images from 196 car classes organized mainly by brand, model, and year.

B. Implementation Details

The experiments combine established few-shot similarity modules with a shared convolutional embedding and a cosine branch, while evaluating configurations across repeated testing episodes. BSNet uses two similarity predictions and tunes their loss contributions on CUB-200-2011.

  • Embedding module: The embedding module uses Conv4 for Matching, Prototype, Relation, and Cosine Networks, and Conv64F for DN4.Conv4 contains four convolution blocks with 3 × 3 convolutions, batch normalization, and ReLU activations.
  • Similarity modules: The Prototype module classifies queries using Euclidean distance to class prototypes formed from one or five support features.For five-shot classification, each class prototype is the mean of its five support features.
  • Similarity modules: The Relation module uses two convolution blocks followed by two fully connected layers, with ReLU and sigmoid activations.The convolution blocks use 3 × 3 convolutions, batch normalization, ReLU, and 2 × 2 max-pooling.
  • Similarity modules: The Image-to-Class module divides Conv64F outputs into 441 local 64-dimensional features and applies cosine measurement with K-nearest neighbors.The embedding output size is 64 × 21 × 21.
  • Bi-similarity module: BSNet combines a self-designed cosine module with Matching, Prototype, Relation, or Image-to-Class similarity modules.The cosine branch is hγ, while gϕ can be any of the other similarity modules.

C. Comparison with State-of-the-art Methods on Few Shot Classification

BSNet is compared with established metric-based few-shot networks across four fine-grained datasets and under different embedding backbones. It generally improves over the compared methods, especially in one-shot settings, although results vary with the backbone and baseline.

  • State-of-the-art comparison: BSNet exceeds Prototype Network and Relation Network on all four fine-grained datasets.The comparison covers Matching Network, Prototype Network, Relation Network, and DN4.
  • Ablation and combinations: Combining the Relation and Cosine modules consistently achieves state-of-the-art performance across the four datasets in one-shot scenarios.Combining the Cosine module with other networks improves mean accuracy by around 2% to 8%, except for Matching Network.
  • Backbone variation: BSNet performs better than compared methods in most backbone experiments and performs best in five-way one-shot tasks with Conv4 or Conv6.With Conv8, BSNet is worse than Prototype Network and DN4 in specified one-shot and five-shot tasks.
  • Ablation study: Figure 3 reports mean accuracy over 600 testing episodes for Relation Network, Cosine Network, and BSNet across Cars, Dogs, Aircraft, and CUB.The ablation compares the two single-branch alternatives with the proposed two-branch R&C model.
  • Backbone variation: BSNet generally outperforms Relation Network with ResNet-10 or ResNet-18, but Relation Network is better with ResNet-34.The backbone comparison uses Stanford-Cars and CUB-200-2011.

E. Ablation Study on Effectiveness of Bi-Similarity Module

The ablation compares single- and bi-similarity networks across fine-grained few-shot settings. BSNet generally outperforms single-similarity alternatives, and its combined branches reduce similarity-specific bias in predictions.

  • In most cases, BSNet outperforms both Relation Network and Cosine Network across Conv4 and ResNet-10 embedding modules.
  • BSNet combines complementary branch preferences: Relation Network scores higher on one Stanford-Cars class, while Cosine Network scores higher on one CUB class.
  • Similarity-score confusion matrices show BSNet correctly predicts queries in cases where Relation and Cosine branches favor different scores.
  • The results indicate that the bi-similarity design is less biased toward any single similarity measure.

F. Feature Visualization

Feature visualizations and broader metric combinations examine why BSNet can improve few-shot fine-grained classification. The evidence links its performance to more discriminative feature regions and favorable behavior across alternative similarity-module combinations.

  • Grad-CAM visualizations show BSNet consistently uses fewer class-discriminative regions than the compared single-metric and metric-based networks.
  • BSNet remains effective when constructed with alternative pairs of similarity metrics, either exceeding both single-metric networks or outperforming the weaker one.
  • With three or four similarity metrics, BSNet either outperforms all single-similarity networks or falls between the best and worst individual networks.
  • The authors attribute improved discriminative features to an embedding that must satisfy two distinct similarity measures.
  • BSNet's parameter-sharing design uses a shared embedding module followed by a bi-similarity module.
Loading 2011.14311v1…