Source-linked AI summary
Few-shot learning of neural networks from scratch by pseudo example optimization
Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, Naonori Ueda
TL;DR
Training neural networks from only a few examples is difficult because deep networks overfit, while common alternatives rely on extra examples or large pretrained models. The paper transfers predictions from a reference estimator to a target using optimized pseudo examples, and reports outperforming naive target training and reference models in its evaluations.
Problem
Learning neural networks from only a few examples remains challenging because deep networks overfit scarce training data and common approaches require additional examples or large pretrained models.
Source: §1 · p. 1Method
The method trains a reference estimator on limited labeled data and transfers its predictions to a target neural network using pseudo examples optimized alongside model parameters.
Source: §1 · pp. 2–3Results
The proposed method outperformed naive target-network training and the reference models, and was superior or comparable to naive training even when the reference was weaker.
Source: §5.2 · p. 9Takeaways & Limitations
The framework is generic and can be applied to other combinations of reference and target models.
Source: Conclusion · p. 10
Abstract
from arXiv · showhide
In this paper, we propose a simple but effective method for training neural networks with a limited amount of training data. Our approach inherits the idea of knowledge distillation that transfers knowledge from a deep or wide reference model to a shallow or narrow target model. The proposed method employs this idea to mimic predictions of reference estimators that are more robust against overfitting than the network we want to train. Different from almost all the previous work for knowledge distillation that requires a large amount of labeled training data, the proposed method requires only a small amount of training data. Instead, we introduce pseudo training examples that are optimized as a part of model parameters. Experimental results for several benchmark datasets demonstrate that the proposed method outperformed all the other baselines, such as naive training of the target model and standard knowledge distillation.
1 Introduction
Learning neural networks from only a few examples remains difficult because deep networks overfit scarce data and common approaches rely on additional examples or large pretrained models. The paper proposes imitation networks, which transfer a reference model’s predictions to a target using optimized pseudo examples.
1 Introduction
- Deep networks readily overfit when training data are scarce, while common approaches require many unlabeled examples or a model pretrained on extensive labeled data.Source: §1 · p. 1
- The universal approximator theorem implies a neural network can imitate other estimators while retaining neural networks’ representation power.Source: §1 · p. 2
- Imitation networks train a reference model on a small labeled set and transfer its predictions to a target; the authors use GPs for locally smooth predictions.Source: §1 · p. 2
- Pseudo examples augment the training data and are optimized to increase training loss while target parameters decrease it, moving them toward areas where the target is poorly trained.Source: §1 · pp. 2–3
- Fidelity weighting uses reference-model prediction uncertainty to eliminate harmful pseudo examples.Source: §1 · p. 2
2 Related work
Prior approaches to few-shot learning and knowledge distillation often rely on extra examples or pretrained models. This work instead uses optimized pseudo examples and permits black-box reference estimators in neural-network training.
2 Related work
- Semi-supervised, transfer, and few-shot approaches commonly require abundant unlabeled data or a model pretrained on large labeled datasets, whereas this method learns from scratch with few examples.Source: §2 · p. 3
- Standard knowledge distillation transfers knowledge from a large teacher to a smaller student using real training data; this method instead uses an arbitrary black-box estimator and optimized pseudo data.Source: §2 · p. 3
- Inducing points were developed for scalable GP inference; here, pseudo examples are introduced into supervised neural-network training and updated to increase loss while model parameters decrease it.Source: §2 · pp. 3–4
3 Model optimization
The framework transfers a reference model’s predictions to a target network, extending knowledge distillation with pseudo examples and uncertainty-based fidelity weighting.
3.1 Knowledge distillation
- Knowledge distillation trains a target to inherit a pre-trained reference model’s generalization ability by matching predictions while also fitting supervised examples.The reference and target may be neural networks of different sizes; the formulation balances supervised and prediction-matching losses.Source: §3.1 · p. 4
3.2 Proposed loss function
- Imitation networks add pseudo training examples so the target can match reference predictions without requiring real labels for that matching loss.The reference model is first trained on the limited labeled examples; the target also seeks accurate predictions on those examples.Source: §3.2 · pp. 4–5
3.3 Fidelity weighting
- Fidelity weighting assigns example-wise weights based on reference-prediction uncertainty, reducing the influence of pseudo examples with unreliable predictions.The method is valid for reference estimators whose uncertainty can be obtained through Bayesian treatments.Source: §3.3 · p. 5
4 Pseudo example optimization
The method optimizes pseudo examples alongside the target model, moving them toward regions where the target is less trained. It alternates model updates with staged pseudo-example updates and merges to stabilize training.
4 Pseudo example optimization
- Pseudo-example selection is a key part of transferring knowledge to improve performance.The section frames pseudo-example optimization as a consequential selection step.Source: §4 · p. 5
4.1 Inducing point method
- Target parameters decrease imitation loss while pseudo examples increase it, moving examples toward regions where the target model is less well trained.The update draws on adversarial training, but pseudo examples receive soft supervision from the reference model rather than ground-truth labels.Source: §4.1 · pp. 5–6
- Pseudo examples are updated in staged sets and merged after each training step, a procedure intended to make model training more stable.The next set starts as a copy of the current set; examples whose predictions differ from the reference are augmented into the training pool.Source: §4.1 · p. 6
4.2 Algorithm
- The algorithm trains a reference model on a few labeled examples, initializes pseudo examples, then alternates target-model and pseudo-example updates.It repeats these updates for a preset number of epochs and training steps, copying and merging pseudo-example sets between steps.Source: §4.2 · pp. 6–7
5 Experiments
Across MNIST and fashion-MNIST, the proposed method outperformed naive target-network training and the reference models. Qualitative tests also show close imitation of the reference, including with sparsely distributed pseudo examples.
5.1 Qualitative analysis
- Qualitative tests used five labeled examples per class; benchmark evaluations used MNIST and fashion-MNIST with 1–20 labeled examples per class.Test accuracy was averaged across 20 selections of supervised examples.Source: §5.1 · p. 7, §5.2 · pp. 8–9
- With 2,500 densely distributed pseudo examples, the target model imitated the reference predictions almost completely.The paper describes this as an unrealistic high-dimensional setup used to test imitation fidelity.Source: §5.1 · p. 8
- With pseudo examples augmented from 250 to 1,000, the target imitated the reference even when examples were sparse; hard supervision improved fit on supervised examples while largely preserving the boundary.The initial examples were generated from supervised examples using interpolation and Gaussian augmentation.Source: §5.1 · p. 8
5.2 Quantitative evaluations
- The proposed method outperformed naive target training and reference models; with many MNIST examples and a weak reference, it was still superior or comparable to naive training.Pseudo-example optimization improved performance, especially on MNIST; fidelity weighting contributed less than optimization.Source: Table 1 · p. 9, §5.2 · p. 9
Conclusion
The paper proposes training a target neural network to imitate a reference model using a limited number of training examples. It optimizes pseudo examples during target training and describes the framework as applicable to other reference-target model pairings.
Conclusion
- The proposed method trains a target network to imitate a GP reference model trained on limited examples, using optimized pseudo examples during target training.The authors describe the broader framework as applicable to other combinations of reference and target models.Source: Conclusion · p. 10
- A possible extension is training a deep target network from a pre-trained shallow reference, which the authors frame as the inverse of knowledge distillation.They suggest this could provide a new way to pre-train deep neural networks.Source: Conclusion · p. 10
- The pseudo-example optimization method is relatively specific, and sophisticated management of pseudo examples remains to be investigated.Source: Conclusion · p. 10