Source-linked AI summary
Image Deformation Meta-Networks for One-Shot Learning
Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, Martial Hebert
TL;DR
One-shot recognition must learn novel visual concepts from very few labeled examples despite image deformations and missing information. IDeMe-Net combines end-to-end meta-learning with a deformation network that fuses probe and gallery images to generate training examples, achieving state-of-the-art results on miniImageNet and ImageNet 1K Challenge benchmarks.
Problem
One-shot learning seeks to recognize novel visual concepts from one or few labeled examples, where large labeled training sets may be unavailable.
Method
IDeMe-Net uses end-to-end meta-learning with a deformation sub-network that fuses probe and gallery image patches to generate additional examples.
Results
The approach achieves state-of-the-art performance on multiple one-shot learning benchmarks, surpassing competing methods by large margins.
Takeaways & Limitations
Synthesized deformed images augment and diversify one-shot training data while retaining critical semantic information for classifier learning.
Takeaways & Limitations
The gallery image pool is sampled only from base categories and adds no images from novel categories.
Abstract
from arXiv · showhide
Humans can robustly learn novel visual concepts even when images undergo various deformations and lose certain information. Mimicking the same behavior and synthesizing deformed instances of new concepts may help visual recognition systems perform better one-shot learning, i.e., learning concepts from one or few examples. Our key insight is that, while the deformed images may not be visually realistic, they still maintain critical semantic information and contribute significantly to formulating classifier decision boundaries. Inspired by the recent progress of meta-learning, we combine a meta-learner with an image deformation sub-network that produces additional training examples, and optimize both models in an end-to-end manner. The deformation sub-network learns to deform images by fusing a pair of images --- a probe image that keeps the visual content and a gallery image that diversifies the deformations. We demonstrate results on the widely used one-shot learning benchmarks (miniImageNet and ImageNet 1K Challenge datasets), which significantly outperform state-of-the-art approaches. Code is available at https://github.com/tankche1/IDeMe-Net.
1. Introduction
One-shot learning addresses recognition of novel concepts from very few labeled examples, but generic meta-learning ignores image structure. IDeMe-Net learns task-useful image deformations from probe-gallery pairs to augment training data and improve one-shot classifiers.
- One-shot learning builds classifiers for new concepts from one or very few labeled examples, addressing limited data and rare visual concepts.
- Generic meta-learning trains classifiers from sampled few-shot tasks on base categories so they can recognize unseen categories.
- Generic meta-learning often treats images as black boxes, whereas humans recognize objects despite ghosting, stitching, montaging, and partial occlusion.
- Deformed images may retain critical semantic information and act as hard examples for formulating classifier decision boundaries.
- IDeMe-Net linearly fuses probe and gallery image patches, preserving probe content while introducing appearance variations through end-to-end meta-optimization.
- IDeMe-Net augments and diversifies one-shot training images and achieves state-of-the-art performance on ImageNet1K and miniImageNet.
2. Related Work
Prior one-shot learning work includes meta-learning, metric learning, and data augmentation. IDeMe-Net differs by dynamically fusing patches from related real images in the image domain while preserving the probe image's label.
- Meta-Learning: Meta-learning maps few training instances or sets to model parameters in simulated one-shot learning scenarios.
- Metric Learning: Metric-learning methods learn spaces optimized for one-shot recognition, including Siamese, matching, prototypical, and relation networks.
- Data Augmentation: Data augmentation addresses scarce one-shot training images using operations such as noise addition, flipping, rotation, rescaling, transformation, and random cropping.
- Comparison with Related Augmentation: Unlike feature-domain hallucination and random convex mixup, IDeMe-Net learns nonnecessarily-convex patch weights for two real images and preserves the probe image's label.
- Comparison with Related Augmentation: The generated images retain important original patches while remaining visually different, facilitating training of one-shot classifiers.
3. One-Shot Learning Setup
The one-shot setup separates base and novel categories and evaluates classifiers learned from episodic few-shot tasks. A gallery sampled from base data supplies unsupervised images for deformation without adding novel-category images.
- Base and novel category sets are disjoint, and the goal is to learn on base data while generalizing to unseen classes from one or few examples.
- Meta-training and meta-testing sample N-way-m-shot episodes with support set S of N × m images and query set Q of N × q images.
- The support set serves as supervised probe images, while a gallery G is randomly sampled from base data as an unsupervised image pool for generating deformations.
- Because G is sampled only from base data, it does not introduce additional images from novel categories or break the standard one-shot setup.
4. Image Deformation Meta-Networks
IDeMe-Net combines a deformation sub-network that synthesizes images from probe-gallery pairs with an embedding sub-network that constructs a one-shot classifier. The deformation process operates patchwise, while classification uses prototype representations and distance-based probabilities.
- IDeMe-Net contains deformation and embedding sub-networks trained together in an end-to-end architecture.The deformation module synthesizes deformed images, and the embedding module maps images to features before constructing the classifier.
- 4.1. Deformation Sub-network: The deformation sub-network fuses probe and gallery images to generate synthesized images while preserving the probe label.The paper assigns yprobe to Isyn and explicitly enforces ysyn = yprobe during optimization.
- 4.1. Deformation Sub-network: A pair of residual branches produces nine weights for linearly combining corresponding patches from the probe and gallery images.Images are divided into a 3×3 grid, and a fully connected layer outputs the 9-D weight vector.
- 4.1. Deformation Sub-network: For each probe image, sampled gallery images produce multiple synthesized images that form an augmented support set.The procedure samples naug gallery images and generates naug deformed instances per probe.
- 4.2. Embedding Sub-network: The embedding sub-network extracts residual-network features and uses a nonparametric prototype classifier for one-shot prediction.An auxiliary softmax classifier with cross-entropy loss is added to facilitate training.
- 4.3. One-Shot Classifier: Class probabilities are computed from distances between query features and class prototypes, and the predicted class has the highest probability.The distance is Euclidean, and prototypes are computed from the augmented support set.
5. Training Strategy of IDeMe-Net
IDeMe-Net is trained episodically by generating augmented support sets from base-category gallery images, then optimizing separate losses for deformation and embedding. Prototype loss improves deformation for one-shot classification, while cross-entropy trains the embedding on synthesized examples.
- 5.1. Training Loss: Training jointly optimizes deformation and embedding subtasks, using prototype loss for the former and cross-entropy loss for the latter.The one-shot prototype classifier itself has no trainable parameters.
- Meta-training Procedure: Meta-training episodes sample N classes, support instances, and query instances from the base categories before initializing a prototype classifier.The sampled support set S trains the initial classifier used during augmentation.
- Meta-training Procedure: For each selected class, the prototype classifier selects the highest-probability gallery images from the fixed base gallery to form a sampling pool.The pool contains the top ϵ% gallery images according to the class probability.
- Meta-training Procedure: Randomly sampled gallery images are fused with probe images to generate synthesized examples for the augmented support set.The deformation network produces Isyn for each sampled probe-gallery pair.
- Meta-training Procedure: The augmented prototype classifier classifies query features to obtain prototype loss, while a softmax classifier computes cross-entropy on the augmented support set.The two losses update the deformation and embedding subnetworks, respectively.
- 5.1. Training Loss: Cross-entropy loss speeds convergence and improves recognition performance compared with using prototype loss alone.The embedding network is trained directly on the augmented support set, which provides relatively more training instances.
6. Experiments
Experiments on ImageNet 1K Challenge and miniImageNet show that IDeMe-Net consistently improves one-shot recognition, while ablations identify deformation generation, gallery selection, and patch-level fusion as important components.
- Experimental setup: 311-way evaluations use 389 base and 611 novel ImageNet classes, with validation and evaluation splits and results averaged over 5 trials.The study focuses primarily on novel classes and evaluates 311-way m-shot settings.
- Experimental setup: The experiments use ResNet-10 for the deformation and embedding networks, ResNet-50 for an additional embedding comparison, and end-to-end SGD training.Training converges over 100 epochs with batch size 32.
- Benchmark results: 7% higher top-5 accuracy than the prototypical network is achieved by IDeMe-Net when m = 1, 2, 5 using ResNet-10.With m = 10, deformed images still provide a 3.5-point boost over the prototype classifier baseline.
- Benchmark results: IDeMe-Net consistently outperforms all baselines with ResNet-50, while Table 1 reports the best performance on novel ImageNet classes with ResNet-10.The reported metrics are top-1 and top-5 accuracy on novel classes.
- Ablation study: Using cross-entropy and prototype losses together performs best, whereas random or support-set gallery selection provides no improvement and omitting meta-testing augmentation approaches baseline performance.These ablations distinguish the contributions of the losses, gallery selection, and synthesized images.
- Ablation study: Patch-level fusion preserves local information and increases diversity, while image-level fusion reduces diversity and pixel-level fusion disrupts local information.Finer 5 × 5 and 7 × 7 divisions may slightly outperform the proposed 3 × 3 division.
- Visualization and augmentation: Performance improves as the number of synthesized images increases from 0 to 8 and saturates when naug > 8.In t-SNE visualizations, IDeMe-Net samples scatter across class manifolds and tend toward class boundaries, while correcting misleading gallery images toward the correct manifold.
- Benchmark results: IDeMe-Net consistently outperforms state-of-the-art competitors on miniImageNet, validating its effectiveness for one-shot learning tasks.The miniImageNet comparison is reported in Table 4.
7. Conclusion
The paper presents a trained image deformation network that generates additional examples for one-shot learning. Experiments report state-of-the-art performance on multiple one-shot learning benchmarks, surpassing competing methods by large margins.
- IDeMe-Net generates additional examples with a trained image deformation network for one-shot learning.
- The deformation network uses unsupervised gallery images and is trained end-to-end through meta-learning.
- Extensive experiments report state-of-the-art performance on multiple one-shot learning benchmarks.
- The approach surpasses competing methods by large margins.