Source-linked AI summary
Improving Generalization via Scalable Neighborhood Component Analysis
Zhirong Wu, Alexei A. Efros, Stella X. Yu
TL;DR
The paper targets open-set visual recognition, where new categories may have very few examples and fixed-category parametric classifiers are inadequate. It learns semantic-preserving embeddings with NCA and augmented memory, achieving strong ImageNet recognition and more generalizable features for sub-category discovery and few-shot recognition.
Problem
Open-set recognition introduces new categories with few examples, limiting the suitability of fixed-category parametric classifiers and motivating generalizable embeddings.
Method
A deep network learns non-parametric embeddings under the NCA criterion, while augmented memory stores dataset-wide embeddings to scale training.
Results
The method achieves remarkable ImageNet classification performance nearly on par with parametric methods and improves generalization for sub-category discovery and few-shot recognition.
Takeaways & Limitations
Nearest-neighbor recognition can provide strong large-scale visual recognition while the learned embedding supports transfer to new categories and few-shot tasks.
Takeaways & Limitations
The model scales linearly with training-set size and the implementation cannot scale to datasets ten times larger than ImageNet under the stated calculations.
Abstract
from arXiv · showhide
Current major approaches to visual recognition follow an end-to-end formulation that classifies an input image into one of the pre-determined set of semantic categories. Parametric softmax classifiers are a common choice for such a closed world with fixed categories, especially when big labeled data is available during training. However, this becomes problematic for open-set scenarios where new categories are encountered with very few examples for learning a generalizable parametric classifier. We adopt a non-parametric approach for visual recognition by optimizing feature embeddings instead of parametric classifiers. We use a deep neural network to learn the visual feature that preserves the neighborhood structure in the semantic space, based on the Neighborhood Component Analysis (NCA) criterion. Limited by its computational bottlenecks, we devise a mechanism to use augmented memory to scale NCA for large datasets and very deep networks. Our experiments deliver not only remarkable performance on ImageNet classification for such a simple non-parametric method, but most importantly a more generalizable feature representation for sub-category discovery and few-shot recognition.
1 Introduction
The paper addresses the mismatch between fixed-category parametric recognition and open-world settings with new categories and few examples. It learns non-parametric embeddings with scalable NCA and reports strong ImageNet performance plus improved generalization for new categories.
- Open-world recognition introduces new categories with few examples, motivating generalizable feature embeddings rather than new parametric classifiers.
- The method learns deep feature embeddings that preserve semantic neighborhood structure using Neighborhood Component Analysis.NCA computes distances among images and uses class labels to define a classification distribution.
- Augmented memory stores dataset-wide embeddings and is updated after training images are visited, making NCA scalable to large datasets and deep networks.Testing uses a k-nearest-neighbor classifier based on the learned metrics.
- Nearest-neighbor recognition achieves remarkable ImageNet performance, nearly matching parametric methods.
- The learned representation improves generalization to new categories for sub-category discovery and few-shot recognition.
2 Related Works
The related work contrasts category-centric recognition with non-parametric data association and situates the paper among memory-based, metric-learning, and NCA approaches. It emphasizes that prior NCA formulations were computationally difficult to scale.
- Object Recognition: Non-parametric data association treats categories as emergent connections among instances and can transfer knowledge not explicit in labels.
- Learning with Augmented Memory: Augmented-memory methods either use differentiable end-to-end memory or treat memory as a non-parametric representation whose size grows with the data.
- Metric Learning: Metric learning is especially relevant when test classes differ from training classes because inference relies on feature representations rather than a subsequent linear classifier.
- NCA: Prior NCA formulations required computing features for the entire dataset at every optimization step, limiting them mainly to small datasets or dimensionality reduction.
3 Approach
The approach learns a non-parametric image embedding whose neighborhood structure supports classification and transfer to novel categories, while augmented memory makes NCA feasible for large datasets and deep networks.
- Feature embedding framework: The embedding queries a search database by image similarity, enabling closed-set recognition, open-set recognition with novel labels, and unannotated content-based retrieval.The classification process does not require extra model parameters or fine-tuning for novel categories.
- Non-parametric formulation: NCA assigns neighbor-selection probabilities from cosine similarities, with leave-one-out neighborhoods controlled by a temperature scale σ.Features are ℓ2-normalized, and each example cannot select itself as a neighbor.
- Non-parametric formulation: The objective directly optimizes embeddings so examples select same-label neighbors, rather than aligning every category example to a learned parametric prototype.NCA’s learning signal allows some positive examples to have negligible probability when other positives align sufficiently well.
- Computational challenges for learning: Original NCA is computationally impractical because each objective term requires embeddings and gradients across the entire dataset, and triplet-style sampling can still require large batches.The paper therefore introduces approximations that enable mini-batch stochastic gradient descent.
- Learning with augmented memory: Augmented memory stores dataset-wide offline embeddings while online embeddings are learned by backpropagation, allowing approximate NCA updates with memory refreshed during training.The memory is updated once per training epoch for each image and is intended to remain approximately current when the learning rate is small.
- Discussion on Complexity: The NCA module adds about 30% extra GPU time over the backbone, but memory and computation scale linearly with training-set size and limit the current implementation to ImageNet-scale data.For a 128-dimensional embedding, one million images, and batch size 256, the reported NCA storage requirement is 6.5 GB.
4 Experiments
Experiments evaluate the non-parametric embedding in ImageNet classification, sub-category discovery, and few-shot recognition. The method performs competitively in closed-world classification and improves transfer to finer-grained and unseen categories.
- 4.1 Image Classification: At k = 30, our method exceeds the parametric classifier by 1.1% on ResNet34 and 0.7% on ResNet50.At k = 1, it improves over the baseline by 3%.
- 4.1 Image Classification: Lower temperature generally improves performance, but the network fails to converge when temperature is too low, such as σ = 0.02.Feature sizes of 128 and 256 produce similar results, while performance degrades below 64 features.
- 4.2 Discovering Sub-Categories: Most images use around 100 −500 supporting neighbors, fewer than the 1,000 images per ImageNet category.Support-set histograms are reported for density thresholds 0.5, 0.7, and 0.9.
- 4.2 Discovering Sub-Categories: On CIFAR100, NCA features reach 62.32% top-1 accuracy, 8% better than baseline features transferred from CIFAR20.The baseline obtains 54.17% with k-nearest neighbors and 58.66% with a linear classifier.
- 4.2 Discovering Sub-Categories: The learned representation preserves visual similarity for fine-grained categories absent from ImageNet127 training labels.Examples emphasize shared color and pose, while prediction gains are strongest for sub-categories with rich intra-class variation.
- 4.3 Few-shot Recognition: The method is on par with prototypical and relation-based networks for shallow few-shot recognition and achieves state-of-the-art results with a deeper network.The deeper-network result is presented as evidence that suitable metric learning can benefit from increased depth.
5 Summary
The paper presents a scalable non-parametric neighborhood approach that learns CNN embeddings preserving semantic label structure through NCA. Augmented memory enables this method to scale, while experiments show strong ImageNet performance and improved generalization for sub-category discovery and few-shot recognition.
- Augmented memory makes NCA scalable for large datasets and deep neural networks.The method learns embeddings whose distances preserve the semantic structure of categorical labels.