Source-linked AI summary
Instance Credibility Inference for Few-Shot Learning
Yikai Wang, Chengming Xu, Chen Liu, Li Zhang, Yanwei Fu
TL;DR
Few-shot learning must recognize novel categories from extremely limited labeled data, while unlabeled instances can introduce harmful pseudo-label noise. ICI estimates pseudo-label credibility with sparse linear regression, iteratively expands the training set with trustworthy instances, and reports new state-of-the-art results across four benchmarks.
Problem
Few-shot learning has too little labeled data to estimate novel-category distributions reliably, and incorrect pseudo-labels can damage classifiers.
Method
ICI trains a linear classifier, pseudo-labels unlabeled data, ranks instances by sparsity-based credibility, and iteratively retrains with selected instances.
Results
ICI establishes new state-of-the-art results on miniImageNet, tieredImageNet, CIFAR-FS, and CUB under two few-shot settings.
Takeaways & Limitations
Iterative credibility-based selection provides a simple statistical way to exploit unlabeled-instance distribution support in few-shot learning.
Takeaways & Limitations
The method assumes access to unlabeled novel-category data, and the authors note that noisy, outlier, or distribution-mismatched instances remain a concern.
Abstract
from arXiv · showhide
Few-shot learning (FSL) aims to recognize new objects with extremely limited training data for each category. Previous efforts are made by either leveraging meta-learning paradigm or novel principles in data augmentation to alleviate this extremely data-scarce problem. In contrast, this paper presents a simple statistical approach, dubbed Instance Credibility Inference (ICI) to exploit the distribution support of unlabeled instances for few-shot learning. Specifically, we first train a linear classifier with the labeled few-shot examples and use it to infer the pseudo-labels for the unlabeled data. To measure the credibility of each pseudo-labeled instance, we then propose to solve another linear regression hypothesis by increasing the sparsity of the incidental parameters and rank the pseudo-labeled instances with their sparsity degree. We select the most trustworthy pseudo-labeled instances alongside the labeled examples to re-train the linear classifier. This process is iterated until all the unlabeled samples are included in the expanded training set, i.e. the pseudo-label is converged for unlabeled data pool. Extensive experiments under two few-shot settings show that our simple approach can establish new state-of-the-arts on four widely used few-shot learning benchmark datasets including miniImageNet, tieredImageNet, CIFAR-FS, and CUB. Our code is available at: https://github.com/Yikai-Wang/ICI-FSL
1. Introduction
Few-shot learning addresses recognition with extremely limited labeled data, where estimating novel-category distributions and trusting pseudo-labels remain difficult. ICI uses iterative credibility-based selection of unlabeled instances to expand training and reports strong benchmark performance.
- Motivation: Few-shot learning targets recognition of new objects with only one or a few labeled examples per category.The setting transfers knowledge from a labeled base dataset to a disjoint but related novel dataset.
- Motivation: Limited labels make data-distribution estimation difficult without introducing inductive bias.Semi-supervised and transductive variants address this by exploiting unlabeled or test-data distributions.
- Problem: Self-taught learning can exploit unlabeled data, but incorrect pseudo-labels may damage the classifier, making instance-level credibility essential.High classifier confidence alone does not guarantee correct categorization, and noisy or outlier instances can be selected.
- Method: ICI trains a linear classifier, pseudo-labels unlabeled instances, ranks their credibility through sparse linear regression, and iteratively adds the most trustworthy instances.The process progressively updates the classifier until all unlabeled samples are included in the expanded training set.
- Contribution: ICI is presented as a simple statistical approach that exploits unlabeled-instance distribution support for few-shot learning.The paper also re-purposes standard self-taught learning by using ICI for pseudo-labeled instance selection.
- Results: The paper reports new state-of-the-art results on miniImageNet, tieredImageNet, CIFAR-FS, and CUB under two few-shot settings.The stated contribution is empirical validation across four widely used few-shot benchmarks.
2. Related work
Related work uses unlabeled data through semi-supervised, self-taught, and transductive approaches, alongside metric- and meta-learning methods for few-shot recognition. ICI differs by statistically estimating pseudo-label credibility and using only the most confident instances for classifier updates.
- Semi-supervised learning: Semi-supervised learning improves performance with limited labeled data by exploiting larger amounts of unlabeled data.Its approaches include low-density separation, consistency regularization, moving averages, and adversarial perturbation regularization.
- Self-taught learning: Self-taught learning predicts labels for unlabeled instances and selects pseudo-labeled data to update the classifier.Prior variants jointly train on labeled and pseudo-labeled data, use mix-up, label propagation, or feature-space reweighting.
- Positioning: ICI differs from prior self-taught-learning variants by statistically estimating each pseudo-labeled instance's credibility and updating with only the most confident instances.The approach is described as conceptually simple and derived from self-taught learning.
- Few-shot learning: Few-shot learning research includes metric-learning methods that improve distances, class prototypes, or task-specific metrics.Examples include Matching Network, Prototypical Network, and TADAM.
- Few-shot learning with unlabeled data: Few-shot learning with unlabeled data includes semi-supervised and transductive settings that use additional data to model novel-category distributions.LST applies self-taught learning in a meta-learning manner.
3. Methodology
ICI augments few-shot classifiers with unlabeled instances by ranking pseudo-label credibility through a sparse linear-regression formulation. It iteratively selects trustworthy samples, retrains the classifier, and repeats until convergence.
- Problem formulation: Few-shot learning uses support examples, query examples, and unlabeled novel-category instances within episodic N-way-m-shot evaluation.The support set contains N × m labeled images, while query accuracy is averaged across episodes and unlabeled data are also available.
- Self-taught learning: Self-taught learning trains a classifier on support data, pseudo-labels unlabeled instances, and adds selected instances to update the classifier.The updated classifier can then use the expanded training set for improved few-shot classification.
- Self-taught learning: Direct self-taught learning is vulnerable because high-confidence predictions can be wrong, while outliers or distribution-mismatched instances can contaminate classifier updates.This motivates ICI as a systematic noise-reduction procedure.
- Instance Credibility Inference: ICI regresses each labeled or pseudo-labeled instance from features to one-hot class labels using a linear model with incidental parameters γ.The coefficient matrix represents classification, while feature vectors and one-hot labels form the regression inputs and outputs.
- Instance Credibility Inference: The method penalizes the incidental matrix with λ and increases sparsity along a regularization path, causing instance-specific γ rows to vanish sequentially.The penalty encourages row-wise vanishing, and the regularization path can be computed with blockwise descent.
- Inference process: The complete algorithm alternates classifier training, pseudo-label inference, ICI ranking, subset selection, and classifier updating until convergence.At inference, the final classifier predicts labels for the query data.
- Instance Credibility Inference: Pseudo-labeled instances are ranked by the λ value at which their γ rows vanish; the earliest-vanishing instance is treated as most trustworthy.In the toy Figure 2 example, the red-line instance vanishes first and is therefore selected as the most trustworthy sample.
4. Experiments
Experiments evaluate ICI across semi-supervised and transductive few-shot settings, benchmark datasets, classifiers, and ablations. Results show robust improvements, state-of-the-art performance, and benefits from iterative credibility-based selection.
- Experimental setup: Experiments use miniImageNet, tieredImageNet, CIFAR-FS, and CUB with ResNet-12 features and linear classifiers including Logistic Regression and SVM.The transductive setting uses five selected instances per class per iteration until the query pool is included.
- Semi-supervised few-shot learning: With equal unlabeled-data quantities, semi-supervised ICI slightly trails or sometimes exceeds transductive results, while additional unlabeled data generally improves performance.With many unlabeled samples, ICI achieves state-of-the-art results even against methods using larger networks and higher-resolution inputs.
- Transductive few-shot learning: ICI improves basic linear classifiers consistently across four benchmark datasets, especially in 1-shot tasks, and reaches state-of-the-art results against transductive few-shot approaches.The improvement margin remains similar between miniImageNet and tieredImageNet, suggesting robustness across their base–novel class relationships.
- Ablation study: Iterative selection outperforms selecting all samples in one step: ICI(3), selecting three samples per class per iteration, beats ICI(8) in the reported comparison.The analysis attributes this to reducing noisy pseudo-labels by incorporating trustworthy unlabeled data progressively.
- Ablation study: ICI improves most initial classifiers regardless of whether their starting accuracy exceeds 50%, although the effect of initial-classifier quality remains an open question.The experiments run 600 episodes with classifiers spanning different accuracy intervals.
- Ablation study: Dimensionality reduction works best when d ≪n; the reported best accuracy is 66.80% at d = 5, while performance remains robust across most reduction algorithms except MDS.Across algorithms, reported accuracies range from 64.61% for LTSA to 67.7% for SE, with MDS at 59.99%.
5. Conclusion
The paper proposes Instance Credibility Inference (ICI), which selects trustworthy pseudo-labeled instances to augment few-shot training. Experiments report new state-of-the-art results across four benchmark datasets.
- ICI exploits unlabeled-instance distribution support for few-shot learning by selecting trustworthy pseudo-labeled examples.
- The method measures pseudo-label credibility by increasing sparsity in a linear regression hypothesis and ranking instances by sparsity degree.
- ICI establishes new state-of-the-art results on miniImageNet, tieredImageNet, CIFAR-FS, and CUB.