Source-linked AI summary
Learning feed-forward one-shot learners
Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip H. S. Torr, Andrea Vedaldi
TL;DR
Deep learning is poorly suited to one-shot learning because it typically requires large labeled datasets, while common alternatives use generative models or discriminative embeddings. The paper introduces a learnet that predicts a pupil network’s parameters from a single exemplar using feed-forward meta-learning and parameter factorizations. It reports improved character-recognition error and encouraging results in visual tracking, while the naive parameter-prediction formulation is constrained by quadratic output size.
Problem
Deep networks require large labeled datasets and are prone to overfitting, making one-shot learning from a single exemplar challenging and motivating alternative generative or embedding-based methods.
Method
A learnet maps one exemplar to the parameters of a discriminative pupil network, with its meta-parameters trained end-to-end on one-shot tasks and parameter factorizations used for feasibility.
Results
The best single-stream learnet reduced character-recognition error from 37.3% for a shared-parameter siamese network to 28.6%, while tracking results exceeded 60 FPS with fewer failures.
Takeaways & Limitations
Feed-forward prediction can obtain deep discriminative-model parameters from one exemplar for one-shot character recognition and visual tracking.
Takeaways & Limitations
Naive parameter prediction scales poorly because the learnet’s output space grows quadratically with the pupil layer width, making even small regressors infeasible.
Abstract
from arXiv · showhide
One-shot learning is usually tackled by using generative models or discriminative embeddings. Discriminative methods based on deep learning, which are very effective in other learning scenarios, are ill-suited for one-shot learning as they need large amounts of training data. In this paper, we propose a method to learn the parameters of a deep model in one shot. We construct the learner as a second deep network, called a learnet, which predicts the parameters of a pupil network from a single exemplar. In this manner we obtain an efficient feed-forward one-shot learner, trained end-to-end by minimizing a one-shot classification objective in a learning to learn formulation. In order to make the construction feasible, we propose a number of factorizations of the parameters of the pupil network. We demonstrate encouraging results by learning characters from single exemplars in Omniglot, and by tracking visual objects from a single initial exemplar in the Visual Object Tracking benchmark.
1 Introduction
The paper targets one-shot learning, where deep networks’ dependence on large labeled datasets and overfitting make learning from a single example difficult. It proposes a learnet that predicts a deep discriminative model from one exemplar in a feed-forward learning-to-learn framework.
- Motivation: Deep networks rely on large labeled datasets and are prone to overfitting, limiting their suitability for learning new concepts from single examples.One-shot learning is commonly addressed with generative models or discriminative embeddings, but embeddings do not directly learn a model of the new object.
- Contribution: The paper asks whether a full deep discriminative model can be induced from a single supervised example without lengthy optimization.
- Contribution: A learnet predicts the parameters of a second network that recognizes other instances of the exemplar’s object class.The learnet is itself a deep neural network and produces the pupil-network parameters from one exemplar.
- Contribution: The approach learns prior domain knowledge offline by solving millions of small one-shot tasks and back-propagating errors end-to-end.
- Contribution: Unlike generative modeling and exemplar SVMs, the learnet uses feed-forward prediction to obtain final model parameters in one go.
- Scope and significance: The paper demonstrates the method on one-shot character recognition and visual object tracking, and positions it as learning complex discriminative-model parameters in one shot.The authors describe the approach as a “learning to learn” method that generates parameters from external exemplars.
2 One-shot learning as dynamic parameter prediction
The paper formulates discriminative one-shot learning as dynamic parameter prediction: a learnet maps one exemplar to the parameters of a predictor, enabling fast feed-forward adaptation. Factorizations make this feasible by reducing the learnet’s predicted parameter space from quadratic to linear growth in layer width or channels.
- One-shot learning requires domain-specific prior information because a single exemplar may be insufficient for generalization, even with regularization.
- A learnet is a non-iterative function that maps an exemplar z to predictor parameters W, with meta-parameters learned offline across many one-shot tasks.Training evaluates the induced predictor on another exemplar-label pair and optimizes the learnet end-to-end.
- Unlike siamese networks, the model treats the exemplar and test input asymmetrically and uses predicted parameters throughout intermediate linear layers rather than only an inner-product similarity score.
- Naively predicting a fully connected weight matrix is infeasible because the learnet output space grows quadratically with layer width; a 100-by-100 example already requires 1M learnet parameters.
- Factorized linear and convolutional layers constrain prediction to diagonal or channel-wise filters, reducing predicted elements from quadratic to linear growth when input and output channel counts are comparable.For convolutions, the factorized branch predicts f^2d elements instead of f^2dk, while fixed projections M and M′ handle feature mixing.
- The proposed architectures include siamese and single-stream learnets, with dynamic convolutions predicted from one example; induced networks can have different internal representations for the same input.
3 Experiments
The experiments compare learnet variants with siamese and factorized baselines on one-shot character recognition and visual object tracking. Learnets improve OCR error rates and combine competitive tracking with feed-forward speed.
- Experimental setup: Experiments compare learnets against siamese baselines on OCR and visual object tracking, using similar network topologies for fairness.The study also includes unshared and factorized siamese networks to separate architectural asymmetry from filter factorization.
- Architectures: Single-stream learnets predict both intermediate filter parameters and final comparison-layer parameters, whereas siamese learnets predict intermediate shared stream parameters.The single-stream configuration can be viewed as predicting a discriminant function; the siamese learnet predicts an embedding function for comparing two images.
- Character recognition: The OCR task tests whether one exemplar identifies the matching character among 20 images from an unseen alphabet.Training uses background alphabets, while evaluation uses separate evaluation alphabets from Omniglot.
- Character recognition: 28.6% error was achieved by the best single-stream learnet, versus 37.3% for a shared-parameter siamese network on foreign-alphabet character recognition.The two learnet architectures performed best with weighted ℓ1 distance, while alternative architectures did not beat the shared siamese network under Euclidean distance.
- Object tracking: Learnet tracking recovers the performance lost by factorized siamese filters and surpasses the original siamese network without degradation from slimmer prediction networks.The tracking experiments report accuracy and failure counts on VOT 2015.
- Object tracking: Feed-forward learnets track at more than 60 FPS while achieving fewer tracking failures than the compared systems, including the 5-FPS SO-DLT tracker.SO-DLT uses online SGD adaptation, whereas learnets generate the model feed-forward from the initial exemplar.
4 Conclusions
The paper shows that a second network can predict a deep network’s parameters from one exemplar through a single feed-forward computation. It demonstrates this approach in one-shot OCR and visual tracking, while identifying domain adaptation and shared learnets as extensions.
- A learnet can obtain a deep neural network’s parameters through a single feed-forward prediction from a second network.
- The approach is desirable when iterative methods are too slow or large sets of annotated training samples are unavailable.
- The paper demonstrates feed-forward parameter prediction in two demanding one-shot tasks: OCR and visual tracking.
- The results suggest a learning-to-learn research direction based on solving millions of small discriminative problems offline.
- Proposed extensions include domain adaptation and sharing one learnet between different pupil networks.
A Basis filters
The appendix interprets factorized convolution as a composition of pixel-wise projections and a diagonal convolution. This represents general filters as combinations of single-channel basis filters, with predicted filters modifying the basis filters.
- The factorized convolution maps activations through two pixel-wise projections surrounding a diagonal convolution: y = Ax = M'WMx.
- M and M' are pixel-wise projections, while W is a diagonal convolution.
- A diagonal convolution uses single-channel filters, whereas pixel-wise projections use scalar coefficients.
- If x has q channels, y has p channels, and intermediate activations have r channels, the factorization combines these components through r intermediate channels.
- The resulting operation is equivalent to a general convolution whose filter aij combines r single-channel basis filters wk.
- Dynamic-convolution predictions modify the r basis filters in the factorized convolutional layer.
B Additional results on object tracking
The tracking evaluation reports official VOT measures alongside displacement and classification errors, and compares the proposed tracker with 62 VOT 2015 trackers. The feed-forward learnet is among the best despite lacking online updates and temporal constraints.
- Table 3 groups tracking architectures by main-network size and marks the best entry in each column, with best overall entries underlined.
- Displacement error measures the average Euclidean distance between the response peak and the ground truth.
- Classification error measures the likelihood that a random positive pair produces a higher response magnitude than a random negative pair.
- The Accuracy-Robustness plot ranks all 62 trackers from VOT 2015, with better trackers positioned closer to the top-right corner.
- The proposed siamese learnet B+A is among the best despite being a proof of concept without online model updates or temporal constraints.
- The tracker operates in feed-forward mode at at least 60 FPS, whereas MDNet fine-tunes with SGD during tracking at 1 frame per second on a GPU.