Source-linked AI summary
Meta-learning with differentiable closed-form solvers
Luca Bertinetto, João F. Henriques, Philip H. S. Torr, Andrea Vedaldi
TL;DR
Few-shot adaptation is difficult because deep networks commonly require large datasets or costly fine-tuning. The paper meta-learns feature representations with differentiable ridge- and logistic-regression solvers, using Woodbury to exploit small episode sizes, and achieves competitive or superior state-of-the-art performance across three benchmarks.
Problem
Few-shot learning must adapt neural models to new concepts from very few labeled examples, despite the data and computational demands of standard approaches.
Method
The paper uses fast differentiable ridge- and logistic-regression solvers as episode-level base learners within a meta-learning system, with Woodbury-accelerated computation.
Results
The methods achieve competitive or superior state-of-the-art accuracy on Omniglot, CIFAR-100, and miniImageNet.
Takeaways & Limitations
Differentiable closed-form and iterative solvers provide a simple adaptation mechanism that combines episode-specific learning with high efficiency.
Abstract
from arXiv · showhide
Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures. Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent. Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently. In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning. The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data. This requires back-propagating errors through the solver steps. While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage. We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components. Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.
1 INTRODUCTION
Few-shot learning seeks to adapt to new classes from very limited labeled data, while standard neural-network training often requires large datasets. The paper uses differentiable, fast-converging solvers as meta-learning base learners and reports competitive or superior benchmark accuracy.
- Motivation: Few-shot learning targets new classes from only a few labeled examples, unlike neural networks that typically need vast training datasets.Limited labels are costly or scarce in applications such as drug discovery.
- Meta-learning: Meta-learning transfers knowledge across episodes through nested loops, with an episode-level base learner and a meta-level learner.The base learner solves each few-shot task, while meta-training improves learning across episodes.
- Base learners: Nearest-neighbour methods are fast but lack test-time data-dependent adaptation, whereas iterative learners can be computationally costly.The paper frames base-learner choice as a trade-off between speed and adaptation.
- Proposed perspective: Ridge regression provides a simple differentiable closed-form base learner whose Woodbury formulation is efficient when examples are few and features are high-dimensional.The proposed approach backpropagates through these learning problems during meta-training.
- Results: Competitive or superior state-of-the-art accuracy is reported on Omniglot, CIFAR-100, and miniImageNet.The experiments evaluate fast, simple base learners across three few-shot benchmarks.
2 RELATED WORK
Related meta-learning work spans similarity-based, recurrent, gradient-based, memory-augmented, and solver-based approaches. This paper positions fast differentiable solvers as a way to retain episode-specific adaptation while avoiding costly iterative optimization.
- Meta-learning origins: Meta-learning emerged decades ago as a framework for improving learning algorithms through experience across tasks.Early work appeared in the 1980s and early 1990s.
- Similarity-based methods: Similarity-function methods are simple and effective for few-shot classification, including Siamese, matching, and prototypical approaches.These methods interpret learning largely as comparing or propagating information between examples.
- Learned updates: Other methods learn update rules using recurrent networks or gradient-based procedures such as SGD and MAML.MAML backpropagates through gradient descent to optimize learner parameters or hyperparameters.
- Memory-based methods: Memory-augmented models retain and retrieve information from newly encountered episodes to address catastrophic forgetting.Their emphasis is long-range storage and access to episode information.
- Solver-based perspective: The proposed solver-based view uses ridge regression to provide episode-specific parameters with faster convergence than adapted SGD.It combines adaptation beyond nearest neighbours with the efficiency of closed-form learning.
3 METHOD
The method meta-learns a feature representation and solver hyperparameters while adapting an episode-specific predictor from support examples. Ridge regression supplies an efficient closed-form solver, and logistic regression extends the framework with Woodbury-accelerated iterative updates.
- 3.1 META-LEARNING: Few-shot meta-learning separates a shared meta-learner from an episode-level base learner trained on each task’s small support set.Episodes contain inputs and labels for learning and evaluation on unseen classes.
- 3.1 META-LEARNING: The predictor maps CNN embeddings to outputs using episode-specific parameters computed by a learning algorithm from training samples.Held-out episode samples evaluate the resulting predictor.
- 3.1 META-LEARNING: Meta-training parameterizes the feature extractor and solver with shared hyperparameters optimized through expected held-out loss across episodes.The resulting objective enables knowledge transfer rather than retraining each predictor independently.
- 3.2 EFFICIENT RIDGE REGRESSION BASE LEARNERS: Ridge regression adds L2 regularization to least squares while preserving a closed-form solution that can be integrated with automatic differentiation.The regularization strength λ is a learnable base-learner hyperparameter.
- 3.2 EFFICIENT RIDGE REGRESSION BASE LEARNERS: The Woodbury identity replaces an embedding-sized inverse with a sample-sized inverse, making the ridge computation efficient for few-shot episodes.The intermediate matrix scales with the number of samples rather than the embedding dimension.
- 3.2 EFFICIENT RIDGE REGRESSION BASE LEARNERS: Ridge regression supports few-shot classification with one-hot targets, while output scale and bias calibrate predictions for cross-entropy loss.The scale and bias, together with λ, are learned by the outer loop.
- 3.3 ITERATIVE BASE LEARNERS AND LOGISTIC REGRESSION: Logistic regression is implemented through Newton’s method as iteratively reweighted least squares, with each iteration accelerated by Woodbury.The paper uses logistic regression as an illustrative iterative solver for binary few-shot classification.
4 EXPERIMENTS
The experiments evaluate R2-D2 and LR-D2 across three few-shot benchmarks and compare their accuracy, efficiency, and adaptation behavior with existing methods.
- 4.1 FEW-SHOT LEARNING BENCHMARKS: The evaluation covers Omniglot, miniImageNet, and CIFAR-FS, including multi-class and binary few-shot classification settings.CIFAR-FS is introduced as a challenging, low-resolution benchmark designed for fast prototyping.
- 4.2 EXPERIMENTAL RESULTS: R2-D2 achieves superior average accuracy to shallow-architecture state-of-the-art methods on miniImageNet and CIFAR-FS.Across the four problems in Table 1, R2-D2 improves by a relative 4.3% over GNN, the second-best method.
- 4.2 EXPERIMENTAL RESULTS: R2-D2 remains competitive on Omniglot and matches SNAIL in the 5-shot case despite using a shallower architecture.SNAIL is outperformed in the 1-shot case, while its performance drops significantly with a shallow embedding.
- 4.2 EXPERIMENTAL RESULTS: LR-D2 produces competitive binary-classification accuracy, but solving N binary problems per iteration makes it less efficient than R2-D2.With five IRLS iterations, LR-D2 reaches 51.9% and 68.7% on miniImageNet and 65.3% and 78.3% on CIFAR-FS for 1-shot and 5-shot settings, respectively.
- 4.2 EXPERIMENTAL RESULTS: LR-D2 outperforms MAML as the number of iterative base-learner steps increases, while Woodbury-based computation provides a significant speedup.Both methods use iterative learners, but LR-D2 uses Newton’s method through IRLS whereas MAML uses SGD.
5 CONCLUSIONS
The paper incorporates fast closed-form solvers into meta-learning for efficient adaptation to unseen problems. R2-D2 performs per-episode adaptation efficiently and achieves strong few-shot benchmark results.
- Closed-form solvers enable efficient adaptation to unseen learning problems within a meta-learning system.
- R2-D2 is almost as fast as prototypical networks while adapting per episode, between non-adaptive metric learning and costly iterative approaches.
- The Woodbury identity provides computational gains when few high-dimensional samples are available.
- The base learners achieve excellent few-shot results and generalize to episodes containing classes unseen during training.
A EXTENDED DISCUSSION
The discussion positions R2-D2 as a fast, differentiable adaptive learner that addresses limitations of both non-adaptive metric methods and iterative adaptation. Experiments show that explicitly training through adaptation is important for competitive few-shot performance.
- R2-D2 combines per-episode adaptation, fast computation, and strong performance by differentiating through a ridge-regression solver.
- The Woodbury identity makes differentiating through the closed-form solver efficient in the low-data regime.
- LR-D2 extends the strategy beyond ridge regression by solving logistic regression through a short series of weighted least-squares problems.
- The results support explicitly enforcing feature generality during training by back-propagating through the adaptation procedure.
B DIFFERENT GAUSSIAN PRIORS FOR REGULARIZATION
The regularization can be interpreted probabilistically or as Tikhonov regularization, and the paper evaluates a more expressive diagonal prior. The diagonal variant allows parameter-specific scales but does not improve results over scalar regularization.
- The regularization term corresponds to a Gaussian parameter prior or, equivalently, Tikhonov regularization.
- A scalar λI represents an isotropic Gaussian prior on the parameters.
- diag(λ) specifies an axis-aligned Gaussian prior with an independent variance for each parameter.
- The diagonal prior can exploit differing parameter scales, but its results do not improve over a scalar λ.
C BASE LEARNER HYPER-PARAMETERS
Figure 3 examines learning the base learner’s hyper-parameters alongside the CNN. Learning the output-calibration parameter is important, whereas learning λ is comparatively unimportant and diagonal regularization can become unstable.
- Learning α to calibrate R2-D2’s output is very important, while learning λ has little effect on performance.
- The diag(λ) variant does not improve results over scalar λ and performance abruptly deteriorates for λ > 0.01.
- Shaded areas in Figure 3 represent 95% confidence intervals.