Source-linked AI summary
Memory Matching Networks for One-Shot Image Recognition
Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, Tao Mei
TL;DR
One-shot recognition remains difficult because CNNs require large annotated datasets, while fine-tuning with very little data can transfer poorly or overfit. MM-Net matches training and inference by writing support-set features to memory and using a contextual learner to predict CNN parameters for unlabelled images. It reports superior results on Omniglot and miniImageNet, including 99.28% and 53.37% one-shot accuracy in the highlighted tasks.
Problem
One-shot recognition is difficult because standard CNN training requires large annotated datasets, while scarce data can make fine-tuning overfit or transfer poorly.
Method
MM-Net matches training and inference by storing support-set features in memory and using a contextual learner to predict CNN parameters for unlabelled images.
Results
99.28% 5-way Omniglot 1-shot accuracy and 53.37% 5-way miniImageNet 1-shot accuracy were reported, with improvements over the cited competitors.
Takeaways & Limitations
MM-Net provides a unified model across different numbers of shots and categories while computing CNN parameters on the fly for new categories.
Abstract
from arXiv · showhide
In this paper, we introduce the new ideas of augmenting Convolutional Neural Networks (CNNs) with Memory and learning to learn the network parameters for the unlabelled images on the fly in one-shot learning. Specifically, we present Memory Matching Networks (MM-Net) --- a novel deep architecture that explores the training procedure, following the philosophy that training and test conditions must match. Technically, MM-Net writes the features of a set of labelled images (support set) into memory and reads from memory when performing inference to holistically leverage the knowledge in the set. Meanwhile, a Contextual Learner employs the memory slots in a sequential manner to predict the parameters of CNNs for unlabelled images. The whole architecture is trained by once showing only a few examples per class and switching the learning from minibatch to minibatch, which is tailored for one-shot learning when presented with a few examples of new categories at test time. Unlike the conventional one-shot learning approaches, our MM-Net could output one unified model irrespective of the number of shots and categories. Extensive experiments are conducted on two public datasets, i.e., Omniglot and \emph{mini}ImageNet, and superior results are reported when compared to state-of-the-art approaches. More remarkably, our MM-Net improves one-shot accuracy on Omniglot from 98.95% to 99.28% and from 49.21% to 53.37% on \emph{mini}ImageNet.
1. Introduction
The paper addresses one-shot recognition, where standard CNN training needs extensive annotations and fine-tuning can overfit or transfer poorly. MM-Net matches training to inference by combining support-set memory with contextual prediction of CNN parameters for unlabelled images.
- Standard deep CNN optimization requires large annotated datasets and performs poorly when learning new categories from very little data.
- Transfer learning may provide limited benefit when the source task or data differs substantially from the target, while scarce data can cause overfitting during fine-tuning.
- MM-Net trains on a small labelled support set in each batch and recognizes other unlabelled instances from those categories, matching its training and test conditions.
- A memory module compresses and generalizes the support set into slots, then produces holistic outputs using information across the support categories.
- A contextual RNN uses memory slots to predict CNN parameters for unlabelled images, capturing long-term training knowledge and short-term test-category information without fine-tuning.
- The memory provides a common representation for different support-set sizes, enabling one unified model regardless of the number of shots and categories.
2. Related Work
Related work spans data augmentation, transfer learning, deep embedding, meta-learning, parameter prediction, and memory networks. MM-Net extends deep embedding by integrating support-set context through memory and predicting CNN parameters for unlabelled images.
- One-shot learning research includes data augmentation, transfer learning, deep embedding learning, and meta-learning approaches.
- Deep embedding methods learn discriminative representations and may classify unlabelled images by matching them with support examples in a shared embedding space.
- Parameter prediction methods generate or evolve weights for one network from another network, including context-dependent fast weights and prediction from a small number of weights.
- Memory Networks augment neural networks with external memory that can be accessed through read and write controllers.
- MM-Net differs from prior deep embedding work by integrating contextual information across support samples into the embedding architecture through memory.
- Unlike Matching Networks, MM-Net predicts CNN parameters for unlabelled images from contextual information encoded in memory slots rather than learning all CNNs during training.
3. One-Shot Image Recognition
MM-Net encodes a few labelled support images into memory, uses that memory to contextualize image embeddings and generate CNN parameters, and trains under support-set-conditioned recognition. Its mixed training strategy converts varying support-set sizes into common memory slots, enabling one unified model across one-shot scenarios.
- Memory Module: MM-Net encodes the entire support set into memory slots that retain class-specific contextual information for recognizing unseen objects.The memory module represents the support set as key-value slots, with keys storing visual representations and values storing class labels.
- Memory Module: A write controller sequentially updates memory by projecting each support image into a key space, finding its nearest memory key, and aggregating or adding the corresponding class information.Updates depend on whether the nearest slot already has the current image’s class label.
- Contextual Embedding: A read controller conditions support-image embeddings on memory so contextual information across categories can produce more discriminative representations.This replaces independent support-sample embedding with contextual embedding based on the encoded memory.
- Contextual Embedding: The contextual learner transforms memory into CNN parameters through sequential bidirectional LSTM encoding, allowing parameters for unlabelled-image embeddings to be synthesized from the support set.The resulting parameters integrate contextual relations among categories into the embedding architecture.
- Training Procedure: MM-Net trains by recognizing unlabelled images conditioned on each batch’s support set, then performs test-time matching on novel categories without fine-tuning.The model uses a non-parametric matching mechanism whose class probabilities are based on support-conditioned embeddings.
- Training Procedure: Mixed batches with different numbers of shots and categories, represented through common memory slots, train one unified architecture for varying one-shot tasks.This addresses the limitation of models tied to a fixed C-way k-shot setting.
4. Experiment
Experiments on Omniglot and miniImageNet compare MM-Net with established one-shot methods across multiple task settings. MM-Net reports strong accuracy, while analyses examine training-strategy generalization, architectural sensitivity, and learned similarities.
- Experimental setup: MM-Net is evaluated against Siamese Networks, Matching Networks, MANN, Meta-N, MAML, and other baselines on Omniglot and miniImageNet.The experiments use standard C-way k-shot recognition with disjoint unlabelled test images and report mean accuracy with 95% confidence intervals.
- Results on Omniglot: 99.28% and 97.16% are MM-Net’s Omniglot accuracies for 5-way and 20-way 1-shot learning, improving over the best competitor by 0.33% and 0.16%.With 5 shots, the corresponding accuracies increase to 99.77% and 98.93%.
- Results on miniImageNet: 53.37% and 66.97% are MM-Net’s miniImageNet 5-way accuracies for 1-shot and 5-shot learning, improving over MAML by 4.67% and 3.86%.The paper reports these as the highest performance then reported on miniImageNet.
- Training strategy: Mixed C-way k-shot training outperforms uniform training across tested shot settings and avoids retraining for new combinations of categories and shots.The mixed strategy varies C ∈ {2, 3, 4, 5} and k ∈ {1, 2, 3, 4, 5}; Mixed k-shot also beats all uniform strategies.
- Architectural analysis: Changing the contextual learner’s bi-LSTM hidden size from 128 to 1,024 changes accuracy by at most 0.013 on both 1-shot and 5-shot tasks.This reduces sensitivity in selecting the optimal hidden-state size.
- Representation analysis: MM-Net’s similarity matrices show mostly higher intraclass and lower inter-class similarities than Matching Networks in the 5-way 5-shot setting.The comparison uses dot-product similarities between 25 support images and 25 corresponding unlabelled test images.
5. Conclusions
MM-Net trains one-shot recognition as it will be performed at inference, using support-set memory and a contextual learner to adapt CNN parameters for unlabeled images. Experiments on Omniglot and miniImageNet validate the approach, with clear performance improvements over other one-shot methods.
- MM-Net trains on one or very few labeled examples per category and switches support sets across batches to match one-shot inference.This training procedure recognizes unlabeled instances from the support-set categories during optimization.
- A memory module contextually augments support-image embeddings with holistic knowledge across categories in each set.
- A contextual learner sequentially uses memory slots to predict CNN parameters on the fly for unlabeled images.
- Experiments on Omniglot and miniImageNet show clear performance improvements over other one-shot learning techniques.