Source-linked AI summary
ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis
Eu Wern Teh, Terrance DeVries, Graham W. Taylor
TL;DR
Distance metric learning seeks effective image similarity, including retrieval across previously unseen classes, while paired-based NCA computation can grow polynomially with dataset size. This paper revisits ProxyNCA and introduces six enhancements, including proxy assignment probability, low-temperature scaling, Global Max Pooling, and faster-moving proxies. ProxyNCA++ improves average Recall@1 by 22.9 percentage points over original ProxyNCA across four zero-shot retrieval datasets and achieves state-of-the-art results across the four benchmarks.
Problem
NCA loss computation grows polynomially with dataset size, motivating ProxyNCA's proxy-based alternative for computationally efficient metric learning.
Method
The paper revisits ProxyNCA and develops six enhancements, including proxy assignment probability, low temperature scaling, Global Max Pooling, and faster-moving proxies.
Results
22.9 percentage points: ProxyNCA++ improves average Recall@1 over original ProxyNCA across four zero-shot retrieval datasets and achieves state-of-the-art performance across all four benchmarks.
Takeaways & Limitations
Low temperature scaling is performance-critical, Global Max Pooling generally outperforms Global Average Pooling, and faster-moving proxies address small proxy gradients.
Takeaways & Limitations
NCA loss computation grows polynomially with the number of samples, so the original NCA formulation uses random sampling and small batches to speed optimization.
Abstract
from arXiv · showhide
We consider the problem of distance metric learning (DML), where the task is to learn an effective similarity measure between images. We revisit ProxyNCA and incorporate several enhancements. We find that low temperature scaling is a performance-critical component and explain why it works. Besides, we also discover that Global Max Pooling works better in general when compared to Global Average Pooling. Additionally, our proposed fast moving proxies also addresses small gradient issue of proxies, and this component synergizes well with low temperature scaling and Global Max Pooling. Our enhanced model, called ProxyNCA++, achieves a 22.9 percentage point average improvement of Recall@1 across four different zero-shot retrieval datasets compared to the original ProxyNCA algorithm. Furthermore, we achieve state-of-the-art results on the CUB200, Cars196, Sop, and InShop datasets, achieving Recall@1 scores of 72.2, 90.1, 81.4, and 90.9, respectively.
1 Introduction
The paper revisits proxy-based distance metric learning for zero-shot image retrieval and introduces enhancements to make ProxyNCA more competitive. ProxyNCA++ improves average Recall@1 over original ProxyNCA and current state-of-the-art models.
- Distance Metric Learning learns similarity measures between examples, here for retrieving images from previously unseen classes.
- ProxyNCA compares samples with learnable class proxies instead of one another, reducing computation but becoming less competitive than recent DML solutions.
- The paper aligns ProxyNCA with NCA through proxy assignment probability and identifies low temperature scaling as performance-critical.
- Global Max Pooling outperforms Global Average Pooling, while faster-moving proxies address small proxy gradients and complement the other enhancements.
- 22.9 percentage points: ProxyNCA++ improves average Recall@1 over original ProxyNCA across four zero-shot retrieval benchmarks.
2 Related Work
Related DML methods either compare image pairs or estimate class distributions with proxies. The paper motivates ProxyNCA partly through the computational and sampling limitations of paired-based approaches, while connecting its enhancements to prior proxy and pooling methods.
- Contrastive and triplet losses optimize distances or relative distances among image pairs, alongside other paired-based loss functions.
- Paired-based methods face polynomially growing pair counts and randomly sampled examples that may provide less information than carefully chosen samples.
- Semi-hard negative mining can produce very few usable negatives, requiring batches on the order of thousands for effectiveness.
- Proxy-based methods reduce computation by comparing samples with class-distribution representatives such as cluster centroids or proxies.
- Prior work uses cosine distance, layer normalization, and class-balanced sampling, while GAP-related scattering concerns motivate alternative pooling strategies.
3 Methods
ProxyNCA++ revisits ProxyNCA and combines six enhancements targeting proxy assignment, temperature scaling, pooling, normalization, sampling, and proxy updates. The method explains why low temperatures help, favors max-based pooling, and uses faster-moving proxies to address weak proxy gradients.
- ProxyNCA++ extends ProxyNCA with six enhancements: proxy assignment probability, low temperature scaling, class balanced sampling, layer normalization, global max pooling, and fast-moving proxies.The original architecture uses a pretrained backbone, a randomly initialized embedding layer, and randomly initialized proxies.
- ProxyNCA compares samples with learnable class proxies instead of one another, reducing the quadratic batch-comparison cost of standard sample-based training.Its objective attracts each sample to its own proxy and repels it from proxies of other classes.
- ProxyNCA++ maximizes proxy assignment probability, preserving attraction to the correct proxy while reducing assignment probability to proxies of other classes.This differs from ProxyNCA’s objective of maximizing a distance ratio and aligns the proxy formulation with NCA’s assignment-probability motivation.
- About Temperature Scaling: Lower temperature produces a peakier distribution and a more refined synthetic-data decision boundary, but lowering it beyond T = 1/9 reduces test generalization on CUB200.The CUB200 plot reports the highest test-average R@1 at T = 1/9; further lowering increases overfitting to the training set.
- About Global Pooling: Global K-Max Pooling interpolates between GMP at k = 1 and GAP at k = M^2, while lower k values correspond to better CUB200 Recall@1.The reported correlation between k and Recall@1 is -0.98.
- About Fast-Moving Proxies: Proxy gradients are three orders of magnitude smaller than embedding and backbone gradients because of proxy L2-normalization, motivating faster-moving proxies.Ablations report a 1.4pp R@1 boost when fast proxies combine with low temperature scaling and a 2.1pp boost when GMP is added.
4 Experiments
The experiments evaluate ProxyNCA++ on four zero-shot image-retrieval datasets using standardized training, backbone, augmentation, and retrieval protocols. ProxyNCA++ outperforms ProxyNCA and other state-of-the-art methods, while ablations identify low-temperature scaling and Global Max Pooling as especially influential.
- Experimental Setup: ProxyNCA++ is evaluated on CUB200, Cars196, Stanford Online Products, and InShop using zero-shot classes.The datasets differ in class and instance counts, with InShop using separate training, query, and gallery groupings.
- Experimental Setup: The training procedure uses a two-stage train–validate–retrain schedule with hyper-parameter tuning on a held-out validation split.Models are first trained on the first half of the original training set, tuned on the second half, then retrained on the combined set.
- Results: ProxyNCA++ outperforms ProxyNCA and other state-of-the-art methods across all reported categories on all four datasets.A 512-dimensional ProxyNCA++ embedding also beats competing methods in that embedding space except on InShop, where it ties for R@1.
- Ablation Study: Removing low-temperature scaling causes the largest CUB200 R@1 drop at -10.8pt, followed by Global Max Pooling at -3.2pt.Other ablation drops are -2.6pt for Layer Normalization and Class Balanced Sampling, -1.9pt for fast proxies, and -1.1pt for Proxy Assignment Probability.
5 Conclusion
The conclusion presents ProxyNCA++ as an enhanced ProxyNCA model built around low-temperature scaling, pooling changes, and faster-moving proxies. It reports broad gains over ProxyNCA and state-of-the-art results across four zero-shot retrieval benchmarks.
- Conclusion: Low-temperature scaling is performance-critical, while Global Max Pooling generally outperforms Global Average Pooling.The paper also explains why low temperature scaling works.
- Conclusion: Faster-moving proxies address the proxies’ small-gradient issue and synergize with low-temperature scaling and pooling choices.The conclusion describes this as part of the enhanced ProxyNCA++ model.
- Conclusion: 22.9 percentage points: ProxyNCA++ improves average Recall@1 over original ProxyNCA across four zero-shot image-retrieval datasets.The reported improvement is averaged across the four benchmarks.
- Conclusion: ProxyNCA++ achieves state-of-the-art results across all categories on the four benchmark datasets.The conclusion reports this result for CUB200, Cars196, Stanford Online Products, and InShop.
A A comparison with NormSoftMax [36]
ProxyNCA++ is compared with NormSoftMax through distance-function, temperature, proxy-learning-rate, pooling, and fast-proxy choices. The ablations show that pooling and combined design choices affect Recall@1 differently across CUB200 and CARS196.
- ProxyNCA++ uses Euclidean squared distance, whereas NormSoftMax uses cosine distance.
- NormSoftMax performs best with temperature scale T = 1/2 and proxy learning rate 4e−1 in the reported sensitivity studies.
- 2.2pp improvement in CUB200 R@1 results from adding Global Max Pooling to NormSoftMax.
- On CUB200, fast proxies alone decrease NormSoftMax R@1 by 0.6pp, while combining fast proxies with GMP increases it by 0.6pp.
- On CARS196, fast proxies alone improve R@1 by 0.3pp, GMP by 1.1pp, and their combination by 1.0pp.
B Two moon classifier
The two-moon sensitivity experiment trains a two-layer model on synthetic scikit-learn two-moons data. The setup uses a 2-to-100 first layer, ReLU, and a 100-to-2 second layer.
- The synthetic experiment uses scikit-learn’s two-moons data generator with 600 samples, noise 0.3, and random state 0.
- The classifier is a two-layer linear model with an input size of 2 and output size of 100 in its first layer.
- A ReLU unit feeds a second layer with input size 100 and output size 2.
C Regarding crop size of images
Image crop size substantially affects retrieval performance, so ProxyNCA++ is evaluated under two crop-size settings against different state-of-the-art references. The reported outcomes vary across datasets and comparison setups.
- Image crop size can have a large influence on performance.
- With 227 × 227 crops, ProxyNCA++ outperforms SOTA on CARS and SOP, ties on CUB, and underperforms on InShop.
- Because SOTA [32] reports no spread information, direct comparison under the older 227 × 227 setup is difficult.
- With 512-dimensional embeddings and 256 × 256 crops, ProxyNCA++ is compared with the current SOTA method [13].
D Regarding the implementation of baseline
The baseline ProxyNCA implementation follows the original algorithm and uses the same training setup as ProxyNCA++. Its hyperparameter search is limited, leaving open the possibility of better results with a broader sweep.
- The implementation replicates the original ProxyNCA baseline and obtains R@1 49.70 on CUB with temperature scale T=1/3, close to the reported 49.2.
- The baseline and ProxyNCA++ use the same training setup, including splitting the original training set into training and validation halves.
- The baseline hyperparameter search was not extensive, and broader tuning might improve both ProxyNCA and ProxyNCA++.
E Regarding the Global Max Pooling (GMP) vs. Global Average Pooling (GAP)
GMP empirically outperforms GAP across object sizes in CUB, but the paper finds no consistent visual explanation for this advantage. Controlling for object size also disproves the initial hypothesis that GAP fails mainly on small objects.
- GMP consistently outperforms GAP across object sizes in the CUB dataset.Figure 7 summarizes Recall@1 performance by object size relative to image size.
- The authors found no consistent visual evidence explaining why GMP works better than GAP.
- Controlling for object size disproved the hypothesis that GAP fails specifically on small objects.
F Regarding the computation complexity of ProxyNCA++
ProxyNCA++ has inference and training costs comparable to the ProxyNCA baseline under the described setup. Training a batch requires distances from each sample to the proxies followed by cross-entropy computation, yielding O(BK) runtime complexity.
- Inference time for ProxyNCA++ is comparable to a ResNet-50 classifier when using a ResNet-50 backbone.Removing the softmax layer and adding layer normalization have negligible effects on inference time.
- ProxyNCA++ has training time complexity comparable to ProxyNCA, with per-batch runtime complexity O(BK).The computation consists of sample-to-proxy distances and cross-entropy over those distances.