PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Rank Pairwise Alignment Bilinear Network For Few-Shot Fine-Grained Image Classification

Huaxi Huang, Junjie Zhang, Jian Zhang, Jingsong Xu, Qiang Wu

arXiv:1908.01313v3cs.CV

TL;DR

Few-shot fine-grained classification must distinguish subtle differences between categories with limited labeled data. LRPABN aligns support and query features before applying low-rank pairwise bilinear pooling, and experiments report superior performance on four fine-grained datasets.

  • Problem

    Few-shot fine-grained classification remains challenging because models must distinguish subtle category differences with limited training data.

  • Method

    LRPABN aligns support and query features before using low-rank pairwise bilinear pooling to extract comparative second-order features.

  • Results

    The proposed model achieves state-of-the-art performance on four benchmark datasets, with improvements reported across fine-grained datasets and experimental settings.

  • Takeaways & Limitations

    Comparing aligned support-query pairs provides the paper's approach for capturing fine-grained relations under few-shot conditions.

Abstract

from arXiv · show

Deep neural networks have demonstrated advanced abilities on various visual classification tasks, which heavily rely on the large-scale training samples with annotated ground-truth. However, it is unrealistic always to require such annotation in real-world applications. Recently, Few-Shot learning (FS), as an attempt to address the shortage of training samples, has made significant progress in generic classification tasks. Nonetheless, it is still challenging for current FS models to distinguish the subtle differences between fine-grained categories given limited training data. To filling the classification gap, in this paper, we address the Few-Shot Fine-Grained (FSFG) classification problem, which focuses on tackling the fine-grained classification under the challenging few-shot learning setting. A novel low-rank pairwise bilinear pooling operation is proposed to capture the nuanced differences between the support and query images for learning an effective distance metric. Moreover, a feature alignment layer is designed to match the support image features with query ones before the comparison. We name the proposed model Low-Rank Pairwise Alignment Bilinear Network (LRPABN), which is trained in an end-to-end fashion. Comprehensive experimental results on four widely used fine-grained classification datasets demonstrate that our LRPABN model achieves the superior performances compared to state-of-the-art methods.

I. INTRODUCTION

Few-shot fine-grained classification must distinguish similar subcategories despite limited labeled data. LRPABN addresses this with aligned support-query comparisons, low-rank pairwise bilinear pooling, and end-to-end learning.

  • Fine-grained classification is difficult because categories have low intercategory variance, while pose, illumination, and camera distance create high intra-category variation.
  • Limited training data remains a central challenge for fine-grained image classification, including endangered-species settings where large-scale samples are difficult to collect.
  • LRPABN combines an encoder, alignment layer, low-rank pairwise bilinear pooling, and comparator in an end-to-end fine-grained few-shot framework.
  • Pairwise bilinear pooling compares support-query image pairs to capture nuanced relations that self-bilinear pooling on individual images misses.
  • Feature alignment matches pairwise convolved image features before bilinear comparison, and ablations identify the alignment mechanism as crucial to improvement over the baseline.
  • The model achieves state-of-the-art performance on four benchmark datasets, supported by more comprehensive experiments and ablation studies.

II. RELATED WORK

Fine-grained recognition methods use regional or global representations, with global bilinear approaches modeling second-order information from whole-image features. These representations improve discrimination but can incur substantial dimensional and computational complexity.

  • A. Fine-Grained Object Classification: Fine-grained classification methods broadly use regional features that localize discriminative object parts or global features extracted from the whole image.
  • A. Fine-Grained Object Classification: BCNN introduced matrix outer products for second-order fine-grained representations, while iSQRT-COV further improved bilinear modeling with covariance matrices.
  • A. Fine-Grained Object Classification: Second-order models can square feature dimensionality, motivating compact bilinear pooling, low-rank covariance decompositions, and factorized bilinear operations.

B. Generic Deep Few-shot Learning

Generic deep few-shot learning commonly uses meta-learning or learning-to-compare strategies to transfer knowledge from auxiliary tasks or data to few-shot targets. The FSFG formulation applies these ideas to fine-grained images, where labeled support samples guide classification of unlabeled query samples.

  • Meta-learning: Meta-learning trains on auxiliary few-shot tasks that mimic target support-query splits, enabling transfer to target data without overfitting.Each auxiliary task is sampled from an annotated auxiliary dataset, and the learned meta-learner is transferred to the target dataset.
  • Learning to compare: Learning-to-compare methods combine a feature embedding network with a similarity metric, allowing query images to be classified using learned representations.Existing approaches use nearest-neighbor, distance-matrix, or cosine-distance classifiers after optimizing transferable embeddings.
  • FSFG methods: Earlier FSFG methods use self-bilinear or covariance pooling to extract subtle image features, but these operations do not directly extract comparative features between image pairs.The cited methods also face high pooled-feature dimensionality, while annotation-based approaches require costly fine-grained annotations.
  • Proposed direction: The proposed framework introduces low-rank pairwise bilinear pooling and a modified nonlinear comparator for fine-grained few-shot classification.The framework is presented as an alternative few-shot embedding structure tailored to the FSFG setting.
  • FSFG formulation: FSFG models target fine-grained classification with few labeled examples, using a labeled subset to classify an unlabeled subset of fine-grained images.With K labeled images across C categories, the task is described as C-way-K-shot.

B. The proposed LRPABN

LRPABN compares support and query images using pairwise bilinear features, aligns their features before comparison, and applies low-rank approximations to reduce the cost of bilinear representations. The resulting design targets more compact comparative features while addressing redundancy and burstiness in full matrix-outer-product pooling.

  • Framework: LRPABN adds low-rank pairwise bilinear pooling to the encoder and modifies a nonlinear comparator for fine-grained few-shot classification.The framework processes support and query images through an encoder before pairwise comparison.
  • Feature alignment: The alignment layer uses an MLP and feature-alignment loss to improve matching when support and query inputs are poorly aligned.The layer is introduced because mismatched input pairs can limit the gain from pooled comparative features.
  • Pairwise pooling: Pairwise pooling forms comparative features from distinct support and query images rather than applying self-bilinear pooling within one image.The pair comes from support set S and query set Q, while the embedding function is shared.
  • Pairwise pooling: 262,144 dimensions result when 512-channel features are combined by matrix outer product, while fewer than 5% of dimensions are reported as informative.The resulting high-dimensional representation also incurs computational load and can suffer redundancy and burstiness.
  • Low-rank formulation: The low-rank formulation reduces parameter complexity from O(nc^2) to O(nc) by factorizing learned transformations.The final form uses 2nc parameters and applies low-rank approximations to both the bilinear representation and learned transformations.
  • Theoretical motivation: The proposed model learns a global dictionary through factorized bilinear coding, contrasting with the unstable local dictionary of earlier matrix-outer-product pooling.The paper connects this distinction to improved scalability and performance over the previous approach.

2) Feature Alignment Layer:

The feature alignment layer rearranges one image’s feature positions to match another image before pairwise bilinear comparison. Two alignment losses encourage compactly matched feature pairs during training.

  • Feature alignment: The alignment mechanism addresses mismatched encoded features caused by pairwise bilinear pooling operating on different images.Unlike self-bilinear pooling, pairwise pooling does not guarantee spatial correspondence between the compared features.
  • Feature alignment: A learned position transform T rearranges feature-vector positions without changing the vectors themselves.The transformed feature X′ preserves the original feature vectors while changing their positions; T is learned with a shallow neural network.
  • Alignment losses: Alignloss1 minimizes Euclidean distances between two transformed embedded features as a rough descriptor-level alignment.The loss uses the feature encoder’s outputs to approximate alignment between the embedded image descriptors.
  • Alignment losses: Alignloss2 sums embedded features along the channel dimension and minimizes the MSE between the resulting features.The support image is transformed to match the target image, and the support is used as the transformed input in the implementation.
  • Alignment outcome: Training with the alignment losses encourages automatically matched features and produces more compact pairwise bilinear feature pairs than the previous method.The position rearrangement matrix is applied to the support or query feature, with support chosen as the transformed input in the implementation.

3) Comparator:

The comparator learns relations between query images and support classes from pairwise comparative bilinear features. In few-shot episodes, class representations aggregate embedded features from the available support images.

  • Comparator role: Pairwise comparative bilinear features are sent to a comparator that learns relations between query images and support classes.The comparator receives the features after the preceding alignment and pooling layers.
  • Support representation: In one-way-K-shot episodes, each support class is represented by a single image.For multi-way-K-shot settings, class representations are computed from the support images in each class.
  • Support representation: In C̃-way-K-shot episodes, each support-class representation is computed as the sum of embedded features from K images.The representation is formed separately for each query image and support class.
  • Relation scores: The comparator output r_i,j is the relation score between query i and class j.The notation defines the score used to compare each query with each support class.
  • Comparator training: Training uses MSE to regress relation scores toward label similarity for query-support pairs.The loss sums squared deviations across m query features and n support-class features.

4) Model Training:

LRPABN trains the comparator with label-similarity regression and optionally adds two alignment losses. Alignment-loss gradients are back-propagated immediately, so the model is updated twice per iteration.

  • Training objective: The bilinear comparator uses MSE loss to regress relation scores toward whether query and support labels match.The indicator δ(y_i = y_j) equals one for matching labels and zero otherwise.
  • Alignment supervision: LRPABN includes two optional alignment losses, Alignloss1 and Alignloss2.These losses supplement the comparator’s relation-score training objective.
  • Optimization: Alignment-loss gradients are back-propagated immediately, updating the model twice during each training iteration.The first update follows computation of the alignment losses, while the main training process supplies the other update.

C. Network Architecture

The network combines an embedding network, low-rank bilinear pooling layer, and comparator network, and evaluates LRPABN across four fine-grained datasets under established split settings.

  • Architecture: The architecture has three parts: Embedding Network, Low-rank Bilinear Pooling Layer, and Comparator Network.These components form the detailed network architecture shown in Figure 3.
  • Embedding Network: The embedding network uses four convolutional blocks with 3 × 3 convolutions, 64 filters, batch normalization, and ReLU layers.The first two blocks additionally use 2 × 2 max-pooling, and the feature alignment layer is integrated into the embedding network.
  • Low-rank Bilinear Pooling Layer: The low-rank bilinear pooling layer applies a 1 × 1 convolution, batch normalization, ReLU, Hadamard product, and normalization.These operations generate the comparative bilinear features.
  • Experimental setting: Table I organizes each dataset by total categories and the auxiliary-versus-target category split used for evaluation.The table defines C_total as the original category count, C_A as auxiliary categories, and C_T as target categories.
  • Comparator Network: The comparator uses two fully connected layers followed by ReLU and Sigmoid nonlinearities to generate relation scores.The relation scores are produced after processing the comparative bilinear features.
  • Experimental setting: Experiments evaluate LRPABN on CUB Birds, DOGS, CARS, and NABirds using dataset splits from PCM and Li’s methods.The four datasets contain 200, 120, 196, and 555 categories respectively; the methods use two groups of established split settings.

B. Experimental Setup

The experiments compare LRPABN variants with generic few-shot and fine-grained few-shot baselines under one-shot and five-shot settings across four datasets.

  • Compared methods: The study evaluates RelationNet, DN4, PCM, CovaMNet, PABN variants, and LRPABN variants on fine-grained few-shot classification.The comparisons include both generic few-shot and FSFG methods.
  • Model variants: PABN and LRPABN variants isolate the effects of alignment losses, low-rank pooling, and the proposed alignment layer.PABN+niv and PABN+cpt use different alignment losses, while LRPABN replaces naive pairwise pooling with low-rank pooling and adds alignment.
  • Datasets and splits: The experiments use four fine-grained datasets with class splits matching prior DN4 and CovaMNet studies.The split distinguishes original, auxiliary-training, auxiliary-validation, and target categories.
  • Evaluation protocol: Both 5-way-1-shot and 5-way-5-shot experiments are conducted with episodic training and testing protocols.The first experiment uses 15 query images during training, while testing uses one query for one-shot and five queries for five-shot episodes.

C. Results and Analysis

Across four fine-grained datasets, LRPABN generally improves classification accuracy over compared methods in one-shot and five-shot settings, while also showing efficiency and qualitative prediction gains.

  • Overall comparisons: LRPABN models achieve significant improvements over state-of-the-art FSFG and generic few-shot methods across one-shot and five-shot tasks on all four datasets.The reported accuracies average results over 10,000 testing episodes.
  • Overall comparisons: LRPABN, PABN+, and PABN obtain around 10% to 30% higher classification accuracy than PCM.The comparison supports pairwise bilinear features over PCM’s self-bilinear features for FSFG tasks.
  • Alignment effects: LRPABNcpt improves over PABNcpt by 1.26% in one-shot and 1.23% in five-shot CUB Birds classification.The corresponding PABN+niv gains over PABNniv are 1.64% and 0.37%.
  • Comparison with CovaMNet: On NABirds, LRPABNcpt gains 7.70% over CovaMNet in one-shot and 5.99% in five-shot classification.The models also improve over CovaMNet on CUB Birds and CARS in both settings.
  • Comparison with DN4: DN4 achieves the highest five-shot accuracy on all four datasets, whereas LRPABN is highest on one-shot CUB Birds, CARS, and NABirds.LRPABNcpt is second-highest on five-shot CUB Birds, CARS, and NABirds.
  • Efficiency: LRPABN requires 2.23 ×10^-3 s per query and is approximately seven times faster than DN4.The efficiency comparison uses the stated feature-map and comparator computation costs.
  • Qualitative analysis: LRPABN-512 makes six errors among 25 queries, compared with seven for LRPABN-128, eight for PABN+, and ten for RelationNet.The fixed batch contains one support image and five query images for each of five classes.

D. Ablation studies

Ablation studies examine low-rank pooling, alignment, input resolution, feature dimension, and comparative-feature visualization to characterize LRPABN’s components and behavior.

  • Low-rank pooling: PABNnew outperforms PABNcpt with lower-dimensional features, but its n × c × c transformation tensor increases model size and inference time.LRPABN approximates this tensor to reduce computational cost.
  • Low-rank pooling: LRPABN uses 213K parameters and 2.23 ×10^-3 s per query, compared with 375K parameters and 8.65 ×10^-3 s for PABNcpt.The low-rank approximation retains superior performance while reducing model size and bilinear feature dimension.
  • Alignment mechanism: Under five-shot CUB Birds classification, PABN+cpt reaches 77.19% accuracy versus 76.81% for PABNcpt.The comparison is used to assess the proposed alignment mechanism.
  • Feature dimension: The feature-dimension experiment evaluates dimensions from 16 through 2048 for one-shot and five-shot CUB Birds classification.Accuracy first improves and then drops as the dimension becomes large, while dimension 16 still gives decent performance.
  • Feature visualization: The t-SNE visualizations compare RelationNet and proposed comparative features on five-way five-shot CUB Birds tasks.The reported visualization supports the capacity of low-rank pairwise bilinear features for FSFG tasks.

V. CONCLUSION

The paper proposes LRPABN for few-shot fine-grained classification using low-rank pairwise bilinear pooling and feature alignment, and reports effectiveness across four datasets.

  • Method: LRPABN extracts second-order comparative features from support-query image pairs through low-rank pairwise bilinear pooling.The method is designed for few-shot fine-grained image classification.
  • Method: A feature alignment mechanism matches embedded support image features with query features before comparison.The alignment mechanism is presented as a component of the proposed comparative feature extraction.
  • Conclusion: Experiments on four fine-grained datasets verify the effectiveness of the proposed method.The paper identifies more sophisticated joint support-query transformations as future work.
Loading 1908.01313v3…