Source-linked AI summary
Relational Embedding for Few-Shot Classification
Dahyun Kang, Heeseung Kwon, Juhong Min, Minsu Cho
TL;DR
Few-shot classification must generalize from a few examples to unseen classes without overfitting to irrelevant features. RENet learns relational embeddings using SCR within images and CCA between images, achieving consistent improvements across four standard benchmarks.
Problem
Few-shot classification seeks to classify queries into unseen classes represented by only a few support images, but learned embeddings can overfit irrelevant features and fail to transfer.
Method
RENet combines self-correlational representation within images and cross-correlational attention between images to learn relational embeddings end to end.
Results
RENet achieves state-of-the-art performance with consistent improvements on miniImageNet, tieredImageNet, CUB-200-2011, and CIFAR-FS.
Takeaways & Limitations
Structural correlations between visual features can generalize to unseen object classes and provide a transferable relational prior for few-shot image recognition.
Abstract
from arXiv · showhide
We propose to address the problem of few-shot classification by meta-learning "what to observe" and "where to attend" in a relational perspective. Our method leverages relational patterns within and between images via self-correlational representation (SCR) and cross-correlational attention (CCA). Within each image, the SCR module transforms a base feature map into a self-correlation tensor and learns to extract structural patterns from the tensor. Between the images, the CCA module computes cross-correlation between two image representations and learns to produce co-attention between them. Our Relational Embedding Network (RENet) combines the two relational modules to learn relational embedding in an end-to-end manner. In experimental evaluation, it achieves consistent improvements over state-of-the-art methods on four widely used few-shot classification benchmarks of miniImageNet, tieredImageNet, CUB-200-2011, and CIFAR-FS.
1. Introduction
Few-shot classification must learn embeddings that generalize from few examples to unseen classes without overfitting to irrelevant features. RENet addresses this by learning relational patterns within images and relational attention between images.
- Few-shot classification learns new visual concepts from only a few support examples per unseen target class.
- Embedding functions can overfit to irrelevant features, limiting transfer to classes absent from training.
- RENet treats relational patterns as potentially more generalizable than individual patterns and learns what to observe and where to attend.
- SCR extracts structural patterns within each image by transforming base representations into self-correlation tensors.
- CCA learns adaptive co-attention between images from refined cross-correlation patterns and semantic relations.
- RENet combines SCR and CCA end-to-end, aggregating cross-attended self-correlational representations into embeddings for few-shot classification.
2. Related work
RENet belongs to metric-based few-shot classification, extending embedding-based comparison with self-correlational representation and convolutionally filtered cross-correlational attention. The paper reports state-of-the-art results on four benchmarks and validates both components through ablations.
- Few-shot classification: Metric-based methods learn embeddings whose distances distinguish the relevance of image pairs, whereas optimization-based methods meta-learn rapid updates.
- Cross-correlation: Cross-correlation is widely used to compute matching costs or similarities between feature maps in correspondence-related vision tasks.
- Contributions: RENet introduces SCR to extract transferable structural patterns within an image.
- Contributions: RENet introduces CCA to learn reliable co-attention between images through convolutional filtering.
- Contributions: Experiments on four standard benchmarks report state-of-the-art performance, while ablations validate the effectiveness of the components.
3. Preliminary on few-shot classification
Few-shot classification evaluates query images against classes represented by only a few support examples, typically using episodic training over disjoint class sets. RENet's architecture processes query-support pairs through self-correlation and cross-correlation before producing relational embeddings.
- Few-shot classification trains on classes Ctrain and evaluates on unseen classes Ctest, where each target class has only a few examples.
- An N-way K-shot episode contains K image-label pairs for each of N classes in the support set.
- During training, episodes teach a mapping from support set S and query image Iq to query label yq; testing applies that mapping to unseen support classes.
- RENet architecture: RENet transforms query and support base representations into self-correlation tensors and then self-correlational representations through convolutional processing.
- RENet architecture: Cross-correlation between the image representations is refined, aggregated bidirectionally into co-attention maps, and applied before producing final relational embeddings.
4. Our approach
RENet learns relational embeddings for few-shot classification by modeling structural patterns within images and co-attention between query-support pairs. SCR enhances image representations with self-correlation features, while CCA uses refined cross-correlations to guide attentive pooling and prototype-based classification.
- Overview: RENet combines self-correlational representation and cross-correlational attention modules in an end-to-end architecture for unseen-class generalization.The SCR module models within-image relations, and CCA models relations between query and support images.
- Self-Correlational Representation (SCR): SCR computes neighborhood self-correlations from base feature maps and applies convolutions over correlation dimensions to extract structural patterns.The resulting relational features are combined with the base representation to form the self-correlational representation F.
- Cross-Correlational Attention (CCA): CCA transforms query and support SCRs into compact features, constructs a cross-correlation tensor, and refines unreliable matches with 4D convolutional matching.The refined cross-correlation is converted into co-attention maps for the two images.
- Relational Embedding: CCA-derived spatial attention maps weight SCR features during pooling, producing query and support embeddings conditioned on their paired images.In an N-way K-shot setting, co-attentive pooling generates multiple query and support views based on pairwise contexts.
- Learning Objective: RENet jointly trains anchor-based and metric-based losses, then predicts each query class using the nearest support prototype.The metric-based loss brings query embeddings closer to prototypes of the same class, while inference uses prototype proximity.
5. Experimental results
RENet is evaluated on four few-shot benchmarks using relational modules for within-image structural patterns and between-image co-attention. It achieves strong benchmark performance, with ablations and comparisons supporting the roles of SCR and CCA.
- Evaluation setup: Experiments evaluate RENet on miniImageNet, tieredImageNet, CUB-200-2011, and CIFAR-FS using standard few-shot classification settings.The evaluation follows 5-way 1-shot and 5-way 5-shot comparisons, with ResNet12 as the backbone.
- Benchmark comparison: RENet sets a new state of the art on miniImageNet, CUB-200-2011, and CIFAR-FS in both 5-way 1-shot and 5-way 5-shot settings.On tieredImageNet, it is comparable to DeepEMD while using a smaller backbone than several competing methods.
- Benchmark comparison: RENet evaluates 2,000 5-way 5-shot episodes in 1.5 minutes, compared with 8 hours for DeepEMD on the same machine.The comparison highlights a substantial inference-time difference associated with DeepEMD’s iterative back-propagation.
- Ablation studies: Both SCR and CCA consistently improve classification accuracy on miniImageNet and CUB, while CCA is especially effective on CUB.The authors attribute CCA’s stronger benefit on CUB to geometric consensus across images with small object-variation differences.
- Ablation studies: SCR achieves lower training accuracy but higher validation accuracy than the GAP baseline, indicating stronger generalization to unseen classes.Figure 4 compares their learning curves on CUB-200-2011 after omitting the first 40 epochs.
- Comparison with attention modules: SCR outperforms self-attention methods, while CCA outperforms CAN and other self-attention methods with a reasonable number of additional parameters.The comparison links SCR to learned local self-correlation patterns and CCA to preserving useful cross-image correlation information.
6. Conclusion
The conclusion presents RENet as a relational embedding approach combining SCR and CCA for few-shot classification. Across four benchmarks, the method achieves state-of-the-art performance and supports relational knowledge as a transferable prior.
- Conclusion: RENet combines self-correlational representation and cross-correlational attention to learn relational embeddings for few-shot classification.The method is presented as learning structural correlations between visual features rather than relying only on individual appearance patterns.
- Conclusion: RENet achieves state-of-the-art results on four standard few-shot classification benchmarks.The conclusion identifies improved generalization to unseen object classes as an experimental observation.
- Conclusion: Learning structural correlations between visual features better generalizes to unseen object classes than the discussed self-attention mechanisms.The authors frame relational knowledge as a promising transferable prior for few-shot image recognition.
A. Appendix
The appendix provides additional details and results for RENet.
- Appendix: The appendix contains additional methodological details and experimental results for RENet.
A.1. Alternative derivation of relational embedding
The alternative derivation obtains relational embeddings by directly applying normalized cross-correlation to feature maps, avoiding explicit attention-map construction. Cross-correlation softly aligns query features with support positions before average pooling.
- Relational embeddings q and s are derived from pre-computed co-attention maps in the main formulation.
- The alternative formulation omits explicit attention maps and instead multiplies feature maps by cross-correlation.
- The normalized cross-correlation tensor is reshaped into a 2D matrix for matrix-based computation.
- The query embedding q results from multiplying the transposed correlation matrix by Fq, followed by average pooling.
- This multiplication softly aligns the query feature map using cross-correlation from each support position.
A.2. Comprehensive details on implementation
The implementation specifies optimization schedules, batch construction, and computational comparisons for the relational embedding model. Training jointly optimizes the stated objectives while separable 4D convolutions are selected for efficiency.
- Training uses SGD with momentum 0.9 and learning rate 0.1, with schedules differing between 1-shot and 5-shot models.One-shot models run for 80 epochs; five-shot models run for 60 epochs, with scheduled learning-rate decay.
- Training batches contain 128 samples for the ImageNet family and 64 for CUB and CIFAR-FS when computing Lanchor.
- The training objective jointly optimizes Lmetric and Lanchor from scratch.
A.3. Ablation studies
Additional ablations evaluate the model on CUB and miniImageNet under the 5-way 1-shot setting.
- The supplementary ablation studies focus on CUB and miniImageNet in the 5-way 1-shot setting.
A.3.1 Self-correlation computation with relative vs. absolute neighbors
The ablations examine neighborhood design, model size, attention temperature, local-window size, SCR, and CCA behavior. Results support relative correlations, relational features, adaptive co-attention, and separable matching refinement within the reported experiments.
- Self-correlation computation with relative vs. absolute neighbors: Relative neighborhood correlations outperform absolute correlations and provide a translation-invariant neighborhood space.
- Attention temperature: The temperature γ balances attention-map smoothness and peakiness, while γ = 100 degrades accuracy by making attention scores excessively uniform.
- Local window size: Accuracy steadily increases with larger SCR neighborhood windows, while U = V = 1 already outperforms the GAP baseline.
- Local window size: The model selects U = V = 5 to limit the space-complexity increase proportional to UV.
- Effect of SCR: CCA with SCR captures finer image details than CCA without SCR, indicating that SCR supplies a more reliable representation for CCA.
- Co-attention on multi-object queries: CCA adaptively captures multiple query objects according to each support instance’s context.
- Cross-correlation refinement: The 4D convolutional block filters unreliable initial matches, including spurious matches between visually similar sky regions.