Source-linked AI summary
Negative Margin Matters: Understanding Margin in Few-shot Classification
Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, Han Hu
TL;DR
Few-shot classification must generalize from well-labeled base classes to novel classes with few examples, but conventional positive margins can harm this open-set transfer. The paper introduces negative-margin softmax-based metric learning, analyzes its effects empirically and theoretically, and reports state-of-the-art performance across three standard benchmarks. Its explanation is that negative margins can trade off base-class discriminability against novel-class transferability.
Problem
Few-shot classification requires transferring knowledge from sufficiently labeled base classes to novel classes with only a few labeled examples, while prior metric-learning practice commonly restricts margins to zero or positive values.
Method
The paper integrates a margin parameter into softmax-based metric learning and studies its effects on training and novel classes through intuitive, empirical, and theoretical analysis.
Results
The negative-margin approach achieves state-of-the-art performance on all three standard few-shot classification benchmarks.
Takeaways & Limitations
Appropriate negative margins can improve novel-class few-shot classification by balancing feature discriminability and transferability.
Takeaways & Limitations
The theoretical result assumes monotonic relationships for base-class discriminability and the inter-class discriminability ratio.
Abstract
from arXiv · showhide
This paper introduces a negative margin loss to metric learning based few-shot learning methods. The negative margin loss significantly outperforms regular softmax loss, and achieves state-of-the-art accuracy on three standard few-shot classification benchmarks with few bells and whistles. These results are contrary to the common practice in the metric learning field, that the margin is zero or positive. To understand why the negative margin loss performs well for the few-shot classification, we analyze the discriminability of learned features w.r.t different margins for training and novel classes, both empirically and theoretically. We find that although negative margin reduces the feature discriminability for training classes, it may also avoid falsely mapping samples of the same novel class to multiple peaks or clusters, and thus benefit the discrimination of novel classes. Code is available at https://github.com/bl0/negative-margin.few-shot.
1 Introduction
Few-shot learning seeks to transfer knowledge from well-labeled base classes to novel classes with only a few labeled examples. This paper finds that negative margins, contrary to common large-positive-margin practice, improve novel-class few-shot classification by balancing training-class discriminability with transferability.
- Motivation: Few-shot learning transfers knowledge from sufficiently annotated base classes to novel classes with only a few labeled examples.The motivation is reducing reliance on expensive massive-scale manually labeled training data.
- Background: Metric-based meta-learning learns metrics on base classes and transfers them to novel classes, with softmax or cosine softmax already achieving strong few-shot performance.These methods can learn a linear classifier on novel classes using the transferred metrics.
- Empirical finding: Larger positive margins improve training-class accuracy but hurt novel-class few-shot performance in the open-set setting.The contrasting behavior appears in Fig. 1 for base-class and novel-class accuracy.
- Explanation: Positive margins can map samples from one novel class to multiple base-class peaks or clusters, whereas an appropriate negative margin offers a better discriminability–transferability tradeoff.The paper also theoretically analyzes novel-class discriminability as decreasing with the margin under stated assumptions.
- Contribution: The proposed negative-margin approach is reported to achieve state-of-the-art performance on three widely used few-shot classification benchmarks.This is presented as a main contribution of the paper.
2 Related Work
Few-shot learning methods include gradient-based, hallucination-based, and metric-based approaches. Against close-set margin-learning assumptions, this paper argues that negative margins better accommodate open-set few-shot classification by trading off feature discriminability and transferability.
- Few-Shot Classification: Few-shot learning methods are broadly categorized as gradient-based, hallucination-based, and metric-based.The categories differ in whether they adapt parameters, generate additional examples, or learn transferable distance metrics.
- Gradient-based methods: Gradient-based methods learn task-agnostic initializations, optimizers, or update mechanisms that adapt to new tasks using limited labeled data.Examples include learned initializations, LSTM-based meta-learners, and external-memory update mechanisms.
- Hallucination-based methods: Hallucination-based methods learn image generators from base classes to synthesize additional examples for novel classes.Some transfer modes of variation, while others jointly optimize the hallucinator and classifier.
- Metric-based methods: Metric-based methods learn transferable distances, using approaches such as cosine similarity, class mean embeddings, or a parametric relation module.MatchingNet, ProtoNet, and RelationNet instantiate these alternatives.
- Margin-based metric learning: The paper explicitly allows negative margins and reports state-of-the-art performance on three standard benchmarks.This challenges prior close-set practice that restricted margins to positive values.
3 Methodology
The method applies margin-adjusted softmax losses during metric pre-training, then analyzes how margin changes feature discriminability and transferability from base to novel classes. Negative margins can trade reduced base-class discriminability for improved novel-class discrimination by avoiding fragmented novel-class mappings.
- Training pipeline: The two-stage pipeline pre-trains metrics on abundant base-class data and fine-tunes a classifier on few-shot novel classes.The negative-margin losses are applied during pre-training, using either inner-product or cosine similarity.
- Negative-margin softmax loss: The generalized large-margin softmax loss subtracts margin m from the ground-truth similarity score, with β controlling score concentration.It operates on deep features zi, labels yi, and classifier weights W.
- Discriminability analysis: Increasing the margin makes base-class features more discriminative because inter-class variance rises substantially while intra-class variance changes little.The discriminative function φ is defined as inter-class variance divided by intra-class variance.
- Discriminability analysis: For novel classes, increasing the margin leaves inter-class variance nearly unchanged but greatly increases intra-class variance, reducing feature discriminability.Larger positive margins consequently hurt novel-class few-shot performance, whereas appropriate negative margins can improve the tradeoff between discriminability and transferability.
- Intuitive explanation: Angular visualizations show that larger margins produce thinner, taller base-class clusters with greater center-angle separation, but can map novel-class samples to multiple peaks or clusters.The visualizations use seven base classes and three novel classes on MNIST, with margins ranging from negative to positive.
- Theoretical analysis: Under stated assumptions, the novel-class discriminative function decreases monotonically with margin, and most mini-ImageNet novel-class subsets favor negative margins.Lower same-base-class pair probability corresponds to more diverse novel-class histograms and lower accuracy; most subsets nevertheless favor negative margins.
4 Experiments
Experiments evaluate negative-margin methods across generic, fine-grained, and cross-domain few-shot settings. Appropriate negative margins improve benchmark performance, while analyses examine regularization, feature structure, and robustness across shot counts.
- Experimental setup: The experiments cover generic object recognition, fine-grained classification, and cross-domain adaptation using mini-ImageNet, CUB, and mini-ImageNet →CUB.The evaluation uses multiple backbone networks and reports episodic 1-shot and 5-shot accuracy.
- Benchmark results: Neg-Cosine achieves state-of-the-art 1-shot and 5-shot performance on almost all four mini-ImageNet backbones.Neg-Softmax also achieves competitive results against existing methods.
- Benchmark results: Both Neg-Softmax and Neg-Cosine significantly outperform comparison methods in the mini-ImageNet →CUB cross-domain setting.Neg-Softmax exceeds the Baseline by 3.73% in the reported 5-shot ResNet-18 evaluation.
- Margin analysis: Appropriate negative margins improve Neg-Softmax and Neg-Cosine over standard softmax and cosine softmax across all three benchmarks.The comparison uses validation-class 1-shot and 5-shot accuracy.
- Regularization and feature analysis: Adding negative margin raises mini-ImageNet accuracy by 5.74% in 1-shot and 4.37% in 5-shot settings before further regularization.Combining regularization techniques yields 62.33% 1-shot and 80.94% 5-shot accuracy on novel classes.
- More shots: Margin = −0.3 consistently outperforms margin = 0 from 1-shot through 300-shot validation settings on mini-ImageNet.The experiment studies accuracy as the number of shots increases from 1 to 300.
5 Conclusion
The paper proposes negative-margin softmax for open-set few-shot classification and combines intuitive and theoretical analyses with experiments. It reports state-of-the-art performance on three standard benchmarks and suggests broader open-set applications.
- Conclusion: Negative-margin softmax is proposed for open-set few-shot classification, challenging the restriction that margins should be non-negative.The paper supports the proposal with intuitive explanation and theoretical proof.
- Conclusion: The approach achieves state-of-the-art performance on all three standard few-shot classification benchmarks.The conclusion frames this result as demonstrated through sufficient experiments.
- Future direction: The paper proposes applying negative margins in broader open-set scenarios without restricting the number of novel-class samples.This is identified as future work rather than an evaluated result.
A Instantiations of Negative-Margin Loss
The paper instantiates margin softmax losses by restricting the margin parameter to non-positive values. The negative-margin loss is applied during pre-training in the proposed pipeline.
- Loss instantiations: The negative-margin softmax and negative-margin cosine softmax formulations restrict the margin parameter to m ≤ 0.The formulations otherwise match the cited large-margin softmax and cosine-loss forms, whose original restrictions use m ≥ 0.
- Training pipeline: The proposed negative-margin loss is applied during the pre-training stage of the pre-training and fine-tuning pipeline.The paper states that regular softmax with m = 0 performs well in the alternative pre-training stage discussed later.
B The effects of margin in the fine-tuning stage
The fine-tuning analysis finds limited sensitivity to the margin parameter: 5-shot accuracy is insensitive, while 1-shot accuracy rises marginally as the margin increases.
- Fine-tuning effects: 5-shot validation accuracy is insensitive to margin variation during fine-tuning.The analysis varies the cosine-softmax margin while keeping the pre-training margin at −0.3 and using ResNet-18.
- Fine-tuning effects: 1-shot validation accuracy increases marginally as the fine-tuning margin increases.This behavior differs from the margin effects observed when negative-margin loss is applied during pre-training.
C Relationship between negative margin and label smoothing
Negative margin and label smoothing both relax the mismatch between softmax outputs and binary labels, but they modify different sides of the objective. Their effects diverge: label smoothing improves base-class performance while negative margin benefits transfer to validation classes.
- Both techniques address the difficulty of matching softmax outputs to binary ground-truth labels, but alter different sides of the objective.Label smoothing softens target labels, whereas negative margin modifies the non-ground-truth side of softmax outputs.
- 3.48% and 0.21% base-class accuracy gains follow label smoothing in 1-shot and 5-shot settings, respectively.With ϵ = 0.05, label smoothing improves performance on base classes but does not consistently transfer to validation classes.
- 1.47% and 3.03% validation-class accuracy drops follow label smoothing in 1-shot and 5-shot settings, respectively.By contrast, negative margin lowers performance on validation images of training classes while benefiting both validation-class settings.
- Negative margin trades lower base-class discriminability for improved feature transferability to novel classes.This differs from label smoothing, which tends to improve base-class discriminability while harming novel-class transferability.