Source-linked AI summary
Delta-encoder: an effective sample synthesis method for few-shot object recognition
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Rogerio Feris, Abhishek Kumar, Raja Giryes, Alex M. Bronstein
TL;DR
Few-shot recognition asks how to classify new visual categories from only one or a few examples, despite deep classifiers typically relying on many examples. The paper introduces the Delta-encoder to synthesize novel-category samples from transferable seen-class deformations, reporting state-of-the-art few-shot results and improved benchmark performance.
Problem
Few-shot visual classification seeks to learn new categories from one or a few examples rather than the many examples used by large-scale deep-learning datasets.
Method
The Delta-encoder learns transferable intra-class deformations between seen-class pairs and applies them to few examples of unseen categories to synthesize training samples.
Results
The Delta-encoder achieves state-of-the-art results in few-shot classification and improves performance across standard few-shot benchmarks.
Takeaways & Limitations
Synthesizing samples from a single or few examples provides a method for learning from scarce labeled data without additional related examples or per-instance semantic information.
Takeaways & Limitations
The approach assumes that plentiful seen-class examples provide latent semantic structure transferable to novel-class distributions.
Abstract
from arXiv · showhide
Learning to classify new categories based on just one or a few examples is a long-standing challenge in modern computer vision. In this work, we proposes a simple yet effective method for few-shot (and one-shot) object recognition. Our approach is based on a modified auto-encoder, denoted Delta-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from it. The synthesized samples are then used to train a classifier. The proposed approach learns to both extract transferable intra-class deformations, or "deltas", between same-class pairs of training examples, and to apply those deltas to the few provided examples of a novel class (unseen during training) in order to efficiently synthesize samples from that new class. The proposed method improves over the state-of-the-art in one-shot object-recognition and compares favorably in the few-shot case. Upon acceptance code will be made available.
1 Introduction
The paper targets visual classification from one or a few examples by learning from plentiful seen-category data. Its approach extracts transferable deformations between seen-class examples, applies them to novel categories, and improves few-shot performance on standard benchmarks.
- Few-shot recognition remains difficult for neural systems trained on large datasets with thousands of examples per category.Humans can learn new categories from a few or even a single example, motivating the problem.
- The approach assumes that examples from known classes provide latent structure useful for sampling from new-class distributions.The paper frames this as learning from analogies across categories.
- The method learns transferable non-linear deformations, called deltas, between same-class examples and applies them to few examples of novel categories.This enables synthesis of additional samples for unseen classes.
- Synthesized samples provide enough data to train a classifier in the standard supervised fashion for few-shot categories.
- The proposed method improves over state-of-the-art few-shot classification methods across a variety of standard benchmarks.
2 Related work
Related work includes metric-learning, meta-learning, and diverse sample-synthesis strategies for few-shot recognition. The Delta-encoder belongs to the synthesis family while learning transferable deformations directly from seen-class pairs.
- Metric-learning methods embed examples into a space where new instances are classified by proximity to a few labeled examples.
- Meta-learning methods train on few-shot tasks so models can adapt to new tasks with relatively few examples.
- Sample-synthesis approaches use unlabeled data, pre-trained models, domain transfer, rendering, augmentation, GANs, or semantic information.
- Prior work conjectures that relative linear offsets between same-class examples encode deformations transferable to other classes.
- Another approach jointly optimizes a noise-conditioned generator and meta-learner through classification loss for random augmentation.
3 The ∆-encoder
The Delta-encoder is an auto-encoder-like model that represents transferable intra-class deformations and applies them to novel-class examples. It samples deltas from seen-class pairs, combines them with a novel example, and generates synthetic samples.
- The Delta-encoder synthesizes samples for unseen categories from one or a few real examples, then uses them to train a classifier.
- Its encoder extracts transferable deformations between same-class examples, while its decoder applies them to other examples.
- Unlike a standard auto-encoder’s semantic bottleneck, the model’s representation encodes additional information needed to reconstruct X from anchor Y.
- During training, the encoder receives X and anchor Y, and a low-dimensional Z forces reconstruction to rely on Y.
- During synthesis, deltas Z=E(X_s,Y_s) from same-class seen pairs are applied through D(Z_i,Y_u) to a novel-class example.
- Experiments use pre-computed VGG16 or ResNet18 feature vectors, with a 16-dimensional encoder output and MLP encoder-decoder networks.
4 Results
Across standard few-shot benchmarks, the Delta-encoder synthesizes samples for unseen classes and performs favorably against state-of-the-art methods. Ablations and synthesis analyses show that learned nonlinear deltas, stronger features, and sufficient generated samples contribute to performance.
- Evaluation setup: The method was evaluated on miniImageNet, CIFAR-100, Caltech-256, CUB, APY, SUN, and AWA2 using standard few-shot protocols.Test episodes sample unseen categories and examples, synthesize 1024 samples per category, train a linear classifier, and evaluate on real query samples.
- Standard benchmarks: The Delta-encoder competes favorably with state-of-the-art few-shot classification methods across the evaluated datasets.Competing results are reported from prior work, with external-data usage marked in the comparison table.
- Additional pre-trained features: More than 10% improvement was achieved on CUB using stronger ImageNet features compared with training on a subset of CUB categories.The shared VGG16 feature extractor was pretrained on ImageNet using categories disjoint from the unseen test categories.
- Ablation study: 81.1% average one-shot accuracy was obtained when reconstructing from another same-class sample, compared with 64.4% for the attribute-conditioned denoising autoencoder.The comparison tests replacing class attributes with another instance from the same class.
- Ablation study: A non-parametric latent-delta distribution improved performance by 0.5%, while adding the second input Y improved performance by 2.4%.Removing input noise did not change performance, so the simpler autoencoder was selected.
- Ablation study: A linear offset delta produced approximately 10 points lower accuracy than the Delta-encoder, highlighting the value of learned nonlinear delta encoding.The linear variant transfers same-class shifts using Z = Xs − Ys and X̂ = Yu + Z.
- Synthesis analysis: Performance converged after 512−1024 synthesized samples, and one synthesized sample from a one-shot example outperformed five real examples without synthesis.The authors therefore used 1024 synthesized samples in their experiments.
- Synthesis analysis: Synthesized samples formed non-trivial density structures whose populations were not centered on the seed examples, differing from naive random perturbation.The resulting classifier improved over the nearest-neighbors baseline by 20−30 points in Tables 1 and 2.
5 Summary and Future work
The Delta-encoder generates samples for unseen classes from as little as one example and achieves state-of-the-art few-shot classification results. The paper also identifies broader applications and technical extensions for future research.
- The Delta-encoder generates novel samples from an unseen class using as little as one example.
- The method achieves state-of-the-art results in few-shot classification.
- The authors suggest using generated samples in semi-supervised and active learning settings with scarce labeled data.
- Future directions include iterative sampling and conditioning deltas on the anchor example to control transferable transformations.