PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Few-shot Fine-grained Recognition with Background Suppression and Foreground Alignment

Zican Zha, Hao Tang, Yunlian Sun, Jinhui Tang

arXiv:2210.01439v2cs.CV

TL;DR

FS-FGR must recognize novel fine-grained categories with few labeled samples despite high intra-class variation, low inter-class differences, and background interference. The paper proposes a two-stage weakly supervised framework combining BAS, FOA, and L2L similarity across raw and refined images. On three fine-grained benchmarks, it reports state-of-the-art performance, including 82.27% on CUB-200-2011 in 5-way 1-shot recognition.

  • Problem

    FS-FGR seeks to recognize novel fine-grained categories from limited labeled samples despite high intra-class variance, low inter-class difference, and background interference.

  • Method

    A two-stage end-to-end weakly supervised framework combines BAS for background suppression, FOA for foreground alignment, and L2L for local similarity using raw and refined images.

  • Results

    82.27% accuracy is reported on CUB-200-2011 for 5-way 1-shot recognition, exceeding AGPF by 3.54%.

  • Takeaways & Limitations

    The experiments support background suppression and foreground alignment as effective components for few-shot fine-grained recognition.

  • Takeaways & Limitations

    BAS localizes objects inconsistently, especially small objects in corners, because limited feature-map resolution and unbalanced foreground-background distributions cause imprecise localization.

Abstract

from arXiv · show

Few-shot fine-grained recognition (FS-FGR) aims to recognize novel fine-grained categories with the help of limited available samples. Undoubtedly, this task inherits the main challenges from both few-shot learning and fine-grained recognition. First, the lack of labeled samples makes the learned model easy to overfit. Second, it also suffers from high intra-class variance and low inter-class differences in the datasets. To address this challenging task, we propose a two-stage background suppression and foreground alignment framework, which is composed of a background activation suppression (BAS) module, a foreground object alignment (FOA) module, and a local-to-local (L2L) similarity metric. Specifically, the BAS is introduced to generate a foreground mask for localization to weaken background disturbance and enhance dominative foreground objects. The FOA then reconstructs the feature map of each support sample according to its correction to the query ones, which addresses the problem of misalignment between support-query image pairs. To enable the proposed method to have the ability to capture subtle differences in confused samples, we present a novel L2L similarity metric to further measure the local similarity between a pair of aligned spatial features in the embedding space. What's more, considering that background interference brings poor robustness, we infer the pairwise similarity of feature maps using both the raw image and the refined image. Extensive experiments conducted on multiple popular fine-grained benchmarks demonstrate that our method outperforms the existing state of the art by a large margin. The source codes are available at: https://github.com/CSer-Tang-hao/BSFA-FSFG.

I. INTRODUCTION

FS-FGR combines scarce supervision with low inter-class and high intra-class variation, while background clutter and foreground misalignment further hinder recognition. The proposed framework addresses these issues through weakly supervised background suppression, foreground alignment, and local similarity modeling.

  • Motivation: FS-FGR recognizes novel fine-grained categories from only a few labeled samples, combining few-shot and fine-grained recognition challenges.Fine-grained datasets are costly to annotate because labeling requires domain-specific knowledge.
  • Motivation: Background similarity can obscure species differences, and manually annotated bounding boxes improve recognition but undermine few-shot learning’s goal of reducing annotation burden.The paper reports a CUB-200-2011 comparison showing the significance of removing background influence.
  • Motivation: Pose and viewpoint changes create foreground misalignment between support and query images, making semantically relevant local-region alignment non-trivial.The stated goal is to remove cluttered backgrounds and align relevant foregrounds using only image-level labels and limited training images.
  • Proposed framework: The method uses a two-stage weakly supervised framework that is trained end-to-end with only image-level labels.Its modules are a feature extractor, BAS, FOA, and L2L similarity metric.
  • Proposed framework: BAS suppresses background disturbance, FOA aligns support features to query features, and L2L measures local similarity to capture subtle differences.Raw and BAS-refined images are both used, with shared stage structure except for global classifiers and integrated inference similarities.

II. RELATED WORK

Prior work advances few-shot and fine-grained recognition through metric learning, optimization, augmentation, feature encoding, and part localization. However, existing FS-FGR methods often overlook background and pose-related variation, motivating explicit suppression and alignment.

  • Few-shot Learning: Few-shot learning methods include metric-based, optimization-based, and augmentation-based approaches, while metric methods jointly emphasize feature extraction and similarity measurement.These approaches map images into embedding spaces or adapt models for new tasks with few samples.
  • Fine-grained Recognition: Fine-grained recognition distinguishes subordinate categories using local or subtle visual differences, with prior methods relying on annotations or weakly supervised feature and part localization.Examples include bird species, car models, and dog breeds.
  • Few-shot Fine-grained Recognition: Existing fine-grained methods often depend on large-scale annotated datasets, whereas FS-FGR recognizes novel sub-categories from limited labeled samples.This makes FS-FGR a more demanding setting than standard fine-grained recognition.
  • Few-shot Fine-grained Recognition: FS-FGR research has explored bilinear feature encoding, subclassifiers, target-oriented matching, and dual-correlation attention to improve discrimination or feature alignment.These methods address representation learning and relation matching between support-query pairs.
  • Few-shot Fine-grained Recognition: The proposed method differs by combining dense local features, feature alignment, and background suppression in a two-stage framework rather than focusing only on discriminative features or relation matching.The paper identifies pose variations and similar backgrounds as important factors often ignored by existing methods.

III. METHODOLOGY

The method embeds support and query images in episodic few-shot tasks, suppresses background regions, and compares aligned local features. Raw and refined image pathways support matching and localization within the framework.

  • Problem definition: In an N-way K-shot task, the support set contains N categories with K images per category, and query samples are classified from labeled supports.Most experiments use N=5 and K=1 or 5.
  • Problem definition: A shared feature extractor maps each support or query image to a feature map F_i=Θ(x_i) in R^c×h×w for embedding and matching.Support prototypes are formed for nearest-neighbor recognition in the embedding space.
  • The Proposed Framework: The framework extracts raw and refined feature maps, then applies foreground alignment and L2L-based comparison to support query-support matching.The refined stage is generated from the raw-stage input through BAS.
  • The Proposed Framework: BAS generates a class-agnostic activation map, suppresses background activation, and crops and enlarges the predicted foreground region for refined inputs.The refined support and query images are re-encoded by the same backbone.

C. Background Activation Suppression Module

The BAS module suppresses cluttered backgrounds using class-agnostic activation maps and attentive erasing, while FOA aligns support features to query features before local comparison.

  • Background Activation Suppression Module: BAS generates class-agnostic activation maps to disentangle foreground objects from backgrounds without additional annotations beyond image-level labels.The predicted foreground region is localized and cropped into a refined image for subsequent classification.
  • Background Activation Suppression Module: Attentive erasing removes highly activated raw-feature regions before global classification, encouraging exploration of the foreground's full extent.The erased region is selected using an activation threshold.
  • Foreground Object Alignment Module: FOA addresses support-query misalignment caused by foreground pose and position variation by aligning support semantic features with respect to the query.This alignment precedes local-to-local matching of spatial features.
  • Local-to-Local Similarity Metric: The L2L metric compares aligned spatial features through element-wise cosine similarity, producing a local similarity map.It reformulates similarity measurement as local-to-local matching after FOA reduces position deviation.

E. Overall Loss Function

The training objective combines global classification and local few-shot losses, using raw and BAS-refined images; inference relies only on the local branch.

  • Overall Loss Function: The global classification branch trains class-agnostic activation maps using cross-entropy over base-category labels for raw and refined images.This branch constrains foreground prediction and background suppression during training.
  • Overall Loss Function: The local few-shot loss compares raw and refined query features with corresponding support prototypes using local-to-local similarity and nearest-neighbor recognition.Refined prototypes and refined query features are used for the refined local loss.
  • Overall Loss Function: The two stage-weighting factors α and β are both fixed at 0.5 in all experiments, although optimizing them may further improve performance.These factors control contributions from the two stages.
  • Overall Loss Function: The cooperative objective combines global and local losses through a weighting factor λ.The overall formulation is given as Ltotal = Lglobal + λLlocal.
  • Overall Loss Function: At inference, only the local-level few-shot loss branch computes similarity for recognizing query samples.The global classification branch is activated only during training.

A. Datasets

Experiments evaluate the method on CUB-200-2011, Stanford Dogs, and Stanford Cars using established few-shot fine-grained recognition protocols.

  • Datasets: Experiments use three benchmarks: CUB-200-2011, Stanford Dogs, and Stanford Cars, with all input images resized to 84 × 84.The evaluation follows protocols used by prior work.
  • Datasets: CUB-200-2011 has 200 bird species and is split into 100 base, 50 validation, and 50 novel evaluation categories.The dataset contains 11,788 images.
  • Datasets: Stanford Dogs contains 120 dog subclasses split into 70 base, 20 validation, and 30 novel evaluation categories.The dataset contains 20,580 images.
  • Datasets: Stanford Cars contains 196 car models split into 130 base, 17 validation, and 49 novel evaluation categories.The dataset contains 16,185 images; an additional human-bounding-box setting is evaluated for comparison on CUB-200-2011.

B. Implementation Details

The method uses ResNet-12 with a 512×11×11 output feature map and is trained end-to-end from scratch. Experiments report mean accuracy with 95% confidence intervals across few-shot benchmarks.

  • B. Implementation Details: ResNet-12 removes its final pooling layer, producing a 512×11×11 feature map for more accurate suppression and alignment.The backbone has four blocks with 64-128-256-512 filters and introduces no additional parameters.
  • B. Implementation Details: The framework is trained from scratch end-to-end with SGD for 90 epochs, using standard cropping and horizontal-flipping augmentation.The initial learning rate is 0.1, decaying at epochs 60 and 70, with λ set to 0.1.
  • 1) Results on general fine-grained datasets:: 82.27% accuracy on CUB-200-2011 5-way 1-shot exceeds AGPF by 3.54%.The method also improves over TOAN by 16.17% in 1-shot and 8.49% in 5-shot settings.
  • 1) Results on general fine-grained datasets:: On Stanford Dogs, the method exceeds typical FSL methods but trails AGPF by 2.76% in 1-shot and 1.43% in 5-shot.The parameter-free suppression and alignment operations provide an efficiency advantage over AGPF’s additional structures.
  • 1) Results on general fine-grained datasets:: 88.93% and 95.20% accuracy on Stanford Cars are reported for 1-shot and 5-shot settings, respectively, outperforming all compared methods.The corresponding relative gains over leading AGPF and CAN results are 2.03% and 0.41%.

2) Results on CUB-200-2011 with bounding boxes:

On cropped CUB-200-2011, the method establishes state-of-the-art performance, while ablations show complementary benefits from local similarity, foreground alignment, and background suppression. Bounding-box preprocessing further improves the method, especially in 1-shot recognition.

  • 2) Results on CUB-200-2011 with bounding boxes:: The full method exceeds all compared methods on cropped CUB-200-2011 in both 1-shot and 5-shot settings, establishing a new state of the art.With bounding boxes, it surpasses FRN by 3.88% under 1-shot evaluation.
  • 2) Results on CUB-200-2011 with bounding boxes:: Adding bounding-box preprocessing improves the method by 3.73% in 1-shot and 1.77% in 5-shot evaluation.The improvement is attributed to cleaner backgrounds, BAS localization, and FOA learning fine-grained features with different importance.
  • D. Ablation Studies: Using local features consistently outperforms global features across three datasets, supporting local similarity for retaining subtle information.Using only refined images is slightly worse than using raw images, whereas dual raw-and-refined features achieve the best baseline performance.
  • D. Ablation Studies: Attentive erasing improves foreground-map coverage by indirectly increasing foreground activation values.The ablation defines it as erasing regions whose activation exceeds a threshold.
  • D. Ablation Studies: BAS and FOA are complementary: removing BAS from the full model reduces performance, while FOA remains superior to the raw-local baseline without suppression.This comparison evaluates background suppression and foreground alignment separately and jointly.

E. Model Complexity Analysis

The efficiency analysis compares the method with local-feature few-shot learners using parameter and FLOP counts. Although the two-stage design increases computation and storage, the method remains competitive in efficiency.

  • E. Model Complexity Analysis: The analysis reports model parameters and floating-point operations against publicly available local-feature few-shot methods.Smaller parameter and FLOP counts indicate better efficiency.
  • E. Model Complexity Analysis: The two-stage framework doubles computation and storage, yet still achieves competitive efficiency compared with the methods evaluated.

F. Backbones Analysis

Backbone and visualization analyses examine how feature extractors, loss weighting, and the two-stage representation affect localization and embedding quality. Deeper backbones produce cleaner masks, while combining raw and refined stages yields more discriminative and separable features.

  • F. Backbones Analysis: Deeper ResNet-12 produces more compact activation maps, complete masks, and more accurate refined images than shallower backbones.The backbone analysis compares Conv-64, ResNet-12, and ResNet-18 settings.
  • F. Backbones Analysis: The optimal λ is relatively larger for Conv-64 than ResNet-12, indicating that local loss contributes more for the shallower network.The local-loss weighting also delays performance saturation for Conv-64.
  • F. Backbones Analysis: Different α and β weight settings create a significant gap in 1-shot performance, showing sensitivity to raw-versus-refined image contributions.
  • F. Backbones Analysis: BAS localizes objects effectively but performs worse on smaller, distant objects.The attentive-erasing visualization indicates improved foreground-map coverage and localization.
  • F. Backbones Analysis: Refined-stage heatmaps show clearer and more discriminative attended fine-grained cues than raw-stage heatmaps.
  • F. Backbones Analysis: Combining raw and refined stages produces more compact and separable t-SNE clusters than either stage alone.The visualization uses five CUB classes with 30 query images per class under 5-way 1-shot evaluation.

I. Discussion

The discussion finds that background suppression improves FS-FGR performance, while imperfect localization limits further gains. Ground-truth foreground crops substantially exceed the learned suppression setting, highlighting remaining potential and several heuristic design choices.

  • BAS analysis: Background suppression improves both 1-shot and 5-shot performance, but BAS can retain low-activated background regions and localize small corner objects poorly.These limitations make the learned refined image less precise than a ground-truth foreground crop.
  • BAS analysis: Using BAS twice does not further improve performance, suggesting that limited feature-map resolution and foreground-background imbalance constrain localization.The authors identify these factors as possible reasons for imprecise localization.
  • Upper-bound analysis: Ground-truth bounding-box crops surpass all existing methods by a large margin, serving as an oracle upper bound for the proposed framework.The result motivates future work on making BAS approach the quality of ground-truth localization.
  • Overall framework: The framework jointly learns background suppression and foreground alignment end to end using only image-level labels, with local-to-local similarity supporting subtle-difference recognition.The conclusion reports state-of-the-art performance on three fine-grained benchmarks, while noting that some framework designs rely on heuristics.
Loading 2210.01439v2…