Source-linked AI summary
Associative Alignment for Few-shot Image Classification
Arman Afrasiyabi, Jean-François Lalonde, Christian Gagné
TL;DR
Few-shot image classification must recognize novel classes from very few examples, but conventional fine-tuning freezes the feature extractor to limit overfitting. The paper proposes associative alignment, which selects related base classes and aligns their features with novel examples using centroid-based or adversarial objectives. Centroid alignment achieves state-of-the-art performance across object recognition, fine-grained classification, and cross-domain adaptation benchmarks.
Problem
Few-shot image classification requires learning novel classes from very few examples, while freezing the feature encoder during fine-tuning may hinder adaptation of the entire network.
Method
Associative alignment selects base categories similar to novel samples and aligns their feature representations using centroid-based or adversarial strategies.
Results
Centroid alignment significantly outperforms the state of the art across generic object recognition, fine-grained classification, and cross-domain adaptation scenarios.
Takeaways & Limitations
Associative alignment enables effective whole-network training while avoiding overfitting, with centroid alignment generally outperforming previous methods across the evaluated scenarios.
Abstract
from arXiv · showhide
Few-shot image classification aims at training a model from only a few examples for each of the "novel" classes. This paper proposes the idea of associative alignment for leveraging part of the base data by aligning the novel training instances to the closely related ones in the base training set. This expands the size of the effective novel training set by adding extra "related base" instances to the few novel ones, thereby allowing a constructive fine-tuning. We propose two associative alignment strategies: 1) a metric-learning loss for minimizing the distance between related base samples and the centroid of novel instances in the feature space, and 2) a conditional adversarial alignment loss based on the Wasserstein distance. Experiments on four standard datasets and three backbones demonstrate that combining our centroid-based alignment loss results in absolute accuracy improvements of 4.4%, 1.2%, and 6.2% in 5-shot learning over the state of the art for object recognition, fine-grained classification, and cross-domain adaptation, respectively.
1 Introduction
Few-shot classification must adapt to novel classes from very few examples, while standard fine-tuning freezes the feature encoder to avoid overfitting. Associative alignment instead uses similar base classes and aligns them with novel examples, enabling whole-network adaptation and improving benchmark performance.
- Few-shot learning seeks to recognize novel image classes from very few labeled examples.
- Standard transfer learning pre-trains on base classes and fine-tunes on novel examples, but freezing the feature encoder limits adaptation of the entire network.The paper hypothesizes that this restriction hinders performance.
- Associative alignment selects base categories similar to novel samples and aligns their feature representations instead of fine-tuning directly on potentially confusing related bases.The alignment provides additional, more relevant training data while avoiding direct use of mismatched base concepts.
- The paper proposes centroid alignment and adversarial alignment as two strategies for aligning novel and related base classes in feature space.Centroid alignment is more stable but assumes a single-mode class distribution; adversarial alignment avoids that assumption but requires a critic network.
- 4.4% improvement in overall accuracy is reported for 5-shot tieredImageNet classification over the state of the art.The reported centroid-alignment gains also include 1.7% on mini-ImageNet, 2.1% on FC100, 1.2% on CUB, and 6.2% for mini-ImageNet-to-CUB adaptation.
- Prior few-shot work includes meta-learning, transfer learning, regularization, and data augmentation, while this approach borrows related-class structure in feature space rather than interpolating novel examples in image space.The paper distinguishes its single-feature-extractor alignment strategy from random base-example sampling and deformation-based augmentation.
3 Preliminaries
The method starts with base data and a small set of novel-class examples, then uses standard transfer learning with a neural feature extractor and linear classifier. Pre-training updates the network on base categories, whereas conventional fine-tuning updates only the classifier on novel examples.
- Few-shot classification uses a small amount of novel-class data, with examples such as 5-shot or 1-shot settings.
- The model consists of a neural feature extractor f(·|θ), a linear classifier c(x|W), and a scoring function such as softmax.
- During pre-training, the network is trained from scratch on examples from the base categories.
- During conventional fine-tuning, the model adapts to novel classes using novel examples while freezing feature-extractor weights θ and updating classifier weights W.This freezing is described as protection against overfitting when all network weights are updated.
4 Associative alignment
Associative alignment leverages related base categories to expand the effective training data for novel classes while enabling fine-tuning of the entire network. It includes centroid-based and adversarial feature-space alignment strategies.
- 4.1 Detecting the related bases: Related base categories are selected for each novel class and used as additional training examples without modifying their instances.A base–novel similarity matrix ranks base categories, and the highest-scoring B categories are retained.
- 4.1 Detecting the related bases: The similarity matrix M records how often examples from each base class are classified as each novel class, identifying related bases by highest scores.Its dimensions are Kb × Kn, for base and novel class counts respectively.
- 4.2 Centroid associative alignment: Centroid alignment permits fine-tuning all model parameters while reducing overfitting, expanding the usable training pool with coherent related-base instances.Unlike approaches that transform base examples, this method aligns novel examples to related-base distributions directly.
- 4.2 Centroid associative alignment: Centroid alignment pushes novel examples toward the centroid of their related base examples, enforcing intra-class compactness during feature learning.The procedure alternates alignment and classification updates, then updates the entire network using classification loss.
- 4.3 Adversarial associative alignment: Adversarial alignment uses a conditioned critic and Wasserstein-1 distance to match novel and corresponding related-base feature distributions.The critic is optimized before the encoder, followed by classification updates for both related-base and novel examples.
5 Establishing a strong baseline
The paper strengthens standard transfer learning with a different loss and episodic early stopping, then motivates angular-margin classification for compact, separated feature clusters.
- The baseline follows standard transfer learning while incorporating a different loss function and episodic early stopping during pre-training.
- Classification loss functions: Arcmax applies an additive angular margin on the normalized hypersphere to encourage compact, well-separated class clusters.The method adds a geodesic distance margin to distributed examples.
- Classification loss functions: The logit represents the angle between a feature representation and a normalized classifier weight vector.The feature representation is denoted by z, and the classifier weight norm is fixed to one.
- Classification loss functions: The angular-margin loss uses radius s and margin m as hyperparameters while processing N examples.
6 Experimental validation
Experiments across four datasets and multiple backbones evaluate associative alignment against prior methods. Centroid alignment generally improves few-shot accuracy, with especially strong gains on tieredImageNet and cross-domain adaptation.
- Datasets and implementation details: Experiments cover mini-ImageNet, tieredImageNet, FC100, and CUB using Conv4, ResNet-18, and WRN-28-10 backbones.The benchmarks span generic object recognition and fine-grained classification.
- Datasets and implementation details: The evaluation uses two-stage training, data augmentation, Adam optimization, and early stopping during base-category pre-training.The early-stopping window is 50, while Adam uses learning rate 10^-3 and batch size 64.
- mini-ImageNet and CUB with a shallow Conv4 backbone: 1.24% and 2.38% gains over the baseline occur for centroid alignment in 1- and 5-shot mini-ImageNet with Conv4.Centroid alignment outperforms the state of the art in all reported mini-ImageNet Conv4 experiments.
- mini-ImageNet and tieredImageNet with deep backbones: 3.3% and 4.41% gains over MetaOptNet occur on tieredImageNet with ResNet-18, while WRN-28-10 gains are 1.06% and 1.11%.
- Cross-domain evaluation: 6.0% absolute 5-shot improvement over cosmax occurs in mini-ImageNet-to-CUB cross-domain evaluation with centroid alignment.Centroid alignment gains 1.3% and 5.4% over the baseline in 1- and 5-shot settings, respectively.
7 Discussion
Associative alignment enables full-network training while avoiding overfitting, but its related-base selection can include irrelevant examples and its single-mode assumption may limit generalization.
- Discussion: Associative alignment trains the entire network while still avoiding overfitting by using related base categories and feature-space alignment.
- Discussion: The method detects related base categories for each novel class and uses centroid alignment to preserve intra-class structure during classification updates.Adversarial alignment is explored as an alternative.
- Limitations: Approximately 12% of samples become out-of-distribution under centroid nearest-neighbor selection on mini-ImageNet in the stated 5-way 1- and 5-shot ResNet-18 setting.Discarding those samples did not significantly affect classification results.
- Limitations: Multi-modality in some base categories may degrade generalization relative to the single-mode case.
Supplementary Material
The supplementary material documents validation, ablation, visualization, way-count, no-alignment, sensitivity, and margin studies supporting the main experiments.
- The authors and affiliations are Arman Afrasiyabi, Jean-François Lalonde, Christian Gagné, Université Laval, Canada CIFAR AI Chair, and Mila.
- Supplementary analyses: The supplement provides a validation error plot and ablation studies for related-class count, alignment, wrongly related classes, and margin choice.
- Supplementary analyses: It also includes visualizations, experiments with more ways, and comparisons to training without alignment.
1 Validation error plot (refers to sec. 5.2)
Validation error changes with pre-training duration: fixed-length pre-training can overfit base features, after which fine-tuning accuracy declines.
- Validation error plot: Validation accuracy decreases after 150 pre-training epochs in the 1-shot setting, indicating that fixed-epoch pre-training is unsuitable.The experiment uses the cosmax function, pre-trains the entire network on X_b, and fine-tunes only classification weights W on X_n.
- Validation error plot: The plot averages results over 50 episodes and varies the number of pre-training base epochs on mini-ImageNet.The figure examines validation error after fine-tuning as a function of pre-training duration.
2 Ablation study on B (refers to sec. 6.1)
Increasing the number of related base classes generally improves few-shot results, although the adversarial method is an exception in the 5-shot setting.
- Ablation on B: Larger B generally produces better few-shot classification results on mini-ImageNet with a ResNet-18 backbone.The ablation compares three values of B, the number of related base classes selected for each novel class.
- Ablation on B: The adversarial alignment method is the exception to the overall improvement with larger B in the 5-shot scenario.The passage reports this exception specifically within the B ablation.
3 Visualization of the alignment methods (refers to sec. 6.2)
The alignment methods produce comparable visual results with one related base class, while centroid alignment yields more discriminative separation with ten; multimodal base classes remain a limitation.
- Method visualization: With B = 1, centroid and adversarial alignment achieve similar visual results in the 5-shot, 5-way mini-ImageNet setting.The comparison uses t-SNE projections of ResNet-18’s 512-dimensional feature space.
- Method visualization: With B = 10, centroid alignment produces more discriminative class separation than adversarial alignment.The visualization compares the two alignment methods using t-SNE in the same 5-shot, 5-way setting.
- Method visualization: Centroid alignment assumes a single-mode class distribution, so multimodal base categories may degrade its generalization performance.Adversarial alignment avoids this assumption but has greater training complexity because it uses a critic network.
4 More-way (refers to sec. 6.2)
Associative alignment improves performance across more-way mini-ImageNet settings and is relatively stable to imperfect related-class selection, while alignment losses remain important for ResNet-18.
- More-way classification: Associative alignment outperforms the best compared method by 6.67%, 4.47%, and 3.82% in 5-, 10-, and 20-way 5-shot classification, respectively.The experiments use B values of 10, 5, and 3 for the 5-, 10-, and 20-way settings.
- Alignment-loss ablation: Removing alignment losses improves Conv4 accuracy by 0.82% and 0.24% in 1-shot and 5-shot settings, but reduces ResNet-18 accuracy by 2.13% and 2.34%.The comparison evaluates training with novel and related classes without alignment losses against the baseline.
- Selection sensitivity: Replacing five selected related base classes with random classes still improves 5-shot performance, indicating stability to small selection changes.The study keeps B = 10 and reports substantially better performance than replacing all ten selected classes.
- Margin sensitivity: The margin m is adjusted according to architecture rather than dataset in the reported experiments.The authors attribute this adjustment to the margin’s relation to network learning capacity.