Source-linked AI summary
Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, Yi Yang
TL;DR
Few-shot classifiers must generalize from scarce labeled data, and episodic meta-learning does not eliminate the low-data difficulty in novel tasks. TPN learns an end-to-end, task-aware graph to propagate labels across the entire query set, achieving state-of-the-art results on benchmark datasets. Its label propagation uses small episodic graphs, though matrix inversion can become inefficient as the episode size grows.
Problem
Few-shot learning remains challenging because novel tasks contain too few labeled examples, while conventional classifiers can overfit and graph construction is often not task-aware.
Method
TPN jointly learns feature embedding and graph construction episodically, then propagates support labels across the union of support and query instances for transductive prediction.
Results
TPN achieves state-of-the-art results on miniImageNet and tieredImageNet, while its semi-supervised adaptation outperforms other semi-supervised few-shot learning baselines.
Takeaways & Limitations
Learning the propagation graph within episodic meta-learning enables explicit transductive inference for few-shot classification.
Takeaways & Limitations
Matrix inversion takes O(n^3) time, although the episode sizes used in this setting are small.
Abstract
from arXiv · showhide
The goal of few-shot learning is to learn a classifier that generalizes well even when trained with a limited number of training instances per class. The recently introduced meta-learning approaches tackle this problem by learning a generic classifier across a large number of multiclass classification tasks and generalizing the model to a new task. Yet, even with such meta-learning, the low-data problem in the novel classification task still remains. In this paper, we propose Transductive Propagation Network (TPN), a novel meta-learning framework for transductive inference that classifies the entire test set at once to alleviate the low-data problem. Specifically, we propose to learn to propagate labels from labeled instances to unlabeled test instances, by learning a graph construction module that exploits the manifold structure in the data. TPN jointly learns both the parameters of feature embedding and the graph construction in an end-to-end manner. We validate TPN on multiple benchmark datasets, on which it largely outperforms existing few-shot learning approaches and achieves the state-of-the-art results.
1 INTRODUCTION
Few-shot learning remains difficult because scarce labeled examples can produce overfit, high-variance classifiers even with episodic meta-learning. TPN addresses this gap by learning task-aware label propagation over the entire query set and reports strong benchmark performance.
- Few-shot learning seeks classifiers that generalize from only a few labeled examples per class.
- Episodic meta-learning improves generalization by training across diverse classification tasks, but scarce data remains difficult in novel tasks.
- Transductive inference jointly predicts the test set by exploiting relationships between labeled and unlabeled instances.
- TPN learns a label-propagation graph episodically and applies transductive inference to the entire query set.It embeds inputs, constructs a graph from support and query instances, and iteratively propagates support labels.
- TPN explicitly models transductive inference in few-shot learning rather than relying only on indirect information sharing between test examples.
- TPN significantly outperforms naive heuristic-based label propagation methods and achieves state-of-the-art results on miniImageNet and tieredImageNet.Its semi-supervised adaptation also outperforms semi-supervised few-shot learning baselines.
2 RELATED WORK
Related work spans meta-learning, embedding and metric-learning methods, and transductive inference. TPN extends graph-based transduction by learning graph construction within the few-shot meta-learning framework.
- Meta-learning: Meta-learning optimizes over batches of tasks so models can generalize quickly to target few-shot problems.
- Embedding and metric learning approaches: Embedding and metric-learning approaches classify query examples using transferable representations, weighted neighbors, or class prototypes.
- Transduction: Transductive methods use the labeled and unlabeled test data jointly, with graph-based approaches propagating labels through weighted networks.
- Transduction: Prior graph-based transduction methods construct graphs in predefined feature spaces with manually selected hyperparameters.
- Transduction: Nichol et al. share information between test examples through batch normalization, whereas TPN explicitly models the transductive setting.
3 MAIN APPROACH
TPN improves few-shot classification by using transductive inference over the entire query set and learning a task-adaptive graph for label propagation. Its embedding, graph construction, propagation, and loss components are trained jointly through episodic meta-learning.
- 3 MAIN APPROACH: TPN uses the manifold structure of each few-shot task to construct a graph and improve classification through transductive inference.The graph is constructed separately for each episode from the union of support and query instances.
- 3 MAIN APPROACH: The episodic setting samples N classes with K support examples and separate query examples, training predictions on the query set to mimic unseen-class evaluation.Support instances serve as labeled training data, while query instances provide the episode’s prediction objective.
- 3 MAIN APPROACH: TPN combines convolutional feature embedding, example-wise graph construction, label propagation from support to query, and cross-entropy loss generation.The framework uses the same embedding function for support and query instances and jointly trains its parameters end to end.
- 3.2.2 GRAPH CONSTRUCTION: The graph construction module learns an example-wise length scale from each embedded instance, forms a k-nearest-neighbor graph, and applies a normalized graph Laplacian.The learned scale adapts to different tasks, while the graph is built over S ∪ Q.
- 3.2.3 LABEL PROPAGATION: Label propagation initializes support labels in a label matrix and iteratively infers labels for the union of support and query instances using normalized graph weights.A closed-form solution is used to make propagation more efficient in the episodic procedure.
- 3.2.4 CLASSIFICATION LOSS GENERATION: The end-to-end objective computes cross-entropy between propagated predictions and ground-truth labels, jointly updating the embedding and graph-construction parameters.Predicted scores are converted to probabilities with softmax before loss computation.
4 EXPERIMENTS
Experiments evaluate TPN on miniImageNet and tieredImageNet in few-shot, transductive, and semi-supervised settings. TPN achieves strong benchmark performance, with especially large advantages when labeled examples are scarce.
- Datasets: TPN is evaluated on miniImageNet and tieredImageNet, using standard few-shot benchmark datasets.miniImageNet contains 100 classes, while tieredImageNet contains 608 classes with a hierarchical category structure.
- Few-shot learning results: The experiments compare TPN with state-of-the-art methods and transductive baselines, including learned and heuristic label propagation.The comparisons include transductive batch normalization, MAML with transduction, and naive label propagation.
- Few-shot learning results: TPN achieves state-of-the-art results and surpasses competing methods by a large margin, including when trained with regular shots.Performance improves further with Higher Shot training, especially in the 1-shot setting.
- Few-shot learning results: 4.13% is TPN’s absolute improvement over the published state of the art for 1-shot 5-way miniImageNet, versus 1.66% for 5-shot.Across 5-way k-shot experiments from 1 through 10 shots, TPN performs consistently better than other methods.
- Few-shot learning results: TPN’s advantage narrows as the number of shots increases because more labeled data are available.The experiments report the largest margins at lower shot counts, consistent with a reduced transductive bonus when training data increase.
- Semi-supervised comparison: TPN-semi outperforms all other semi-supervised few-shot algorithms by a large margin, especially in the 1-shot case.TPN-semi propagates labels using labeled and extra unlabeled data and can serve when all test data are unavailable.
5 CONCLUSION
TPN frames few-shot learning as transductive inference over the entire test set, using learned embeddings and graph construction for label propagation. It achieves state-of-the-art results on two benchmarks, with stronger performance also reported for its semi-supervised adaptation.
- 5 CONCLUSION: TPN performs transductive few-shot inference through feature embedding, graph construction, label propagation, and loss computation.All parameters are learned end-to-end with cross-entropy loss on query-set predictions.
- 5 CONCLUSION: TPN produces example-wise graph parameters to exploit the manifold structure within each episode.
- 5 CONCLUSION: TPN achieves state-of-the-art results on miniImageNet and tieredImageNet.
- 5 CONCLUSION: Its semi-supervised adaptation achieves higher results than other semi-supervised methods.
- 5 CONCLUSION: Future work will explore episodic-wise distance metrics instead of only example-wise parameters for Euclidean distance.
A ABLATION STUDY
The ablation study examines how training shots and query number affect the model.
- A ABLATION STUDY: The ablation study evaluates the influence of training shots.
- A ABLATION STUDY: The ablation study evaluates the influence of query number.
- A ABLATION STUDY: The study considers query number alongside training shots as experimental factors.
A.1 TRAINING SHOTS
Figure 5 examines model performance as the number of training shots changes, using 5-way test accuracy for 1-shot and 5-shot settings.
- A.1 TRAINING SHOTS: Figure 5 plots the number of training shots on the x-axis.
- A.1 TRAINING SHOTS: Figure 5 reports 5-way test accuracy on the y-axis for 1-shot and 5-shot settings.
- A.1 TRAINING SHOTS: Error bars represent 95% confidence intervals computed over 600 test episodes.
A.2 QUERY NUMBER
The query-number experiments vary training and test queries separately and together. Increasing the test query can improve performance, while performance remains relatively stable across training-query counts when the test query is fixed.
- A.2 QUERY NUMBER: The experiments fix training query to 15, fix test query to 15, or set training query equal to test query.
- A.2 QUERY NUMBER: When training query is fixed, increasing test query improves performance, and even five test queries can yield good performance.
- A.2 QUERY NUMBER: When test query is fixed, performance remains relatively stable across different training-query numbers.
- A.2 QUERY NUMBER: When training and test query numbers match, increasing their number can also improve performance.
A.3 RESULTS ON RESNET
Using a 4-layer neural network for fair comparison, TPN beats TADAM in 1-shot classification and outperforms all other listed recent methods except TADAM in 5-shot classification.
- The comparison uses a 4-layer neural network, whereas several competing methods use 12-layer ResNets.
- TPN beats TADAM in the 1-shot setting.
- For 5-shot classification, TPN outperforms all other recent high-performance methods except TADAM.
A.4 CLOSED-FORM SOLUTION VS ITERATIVE UPDATES
The closed-form label-propagation solution is efficient for the paper’s small few-shot matrices, while iterative optimization performs slightly worse after ten steps because of inaccurate computation and unstable gradients.
- The closed-form solution is efficient for 80 × 80 or 100 × 100 matrices in the paper’s few-shot settings.These dimensions correspond to 5-way, 1-shot, 15-query and 5-way, 5-shot, 15-query configurations.
- 53.05/68.75 accuracy for 1-shot/5-shot experiments was obtained with only 10 iterative optimization steps on miniImageNet.
- The iterative results were slightly worse than the closed-form results, which reached 53.75/69.43 for 1-shot/5-shot experiments.The paper attributes the gap to inaccurate computation and unstable gradients caused by multiple-step iterations.
A.5 ACCURACY WITH 95% CONFIDENCE INTERVALS
The supplied passages identify few-shot accuracy tables for miniImageNet and tieredImageNet, including tables reporting averages over 600 test episodes with 95% confidence intervals.
- Few-shot classification accuracies on miniImageNet are reported in Table 7 with 95% confidence intervals.Results are averaged over 600 test episodes, and top results are highlighted.
- Few-shot classification accuracies on tieredImageNet are reported in Table 8 with 95% confidence intervals.Results are averaged over 600 test episodes, and top results are highlighted.
- Semi-supervised comparisons are listed for miniImageNet in Table 9 and tieredImageNet in Table 10.