Local Similarity-Aware Deep Feature Embedding

Chen Huang; Chen Change Loy; Xiaoou Tang

Source-linked AI summary

Local Similarity-Aware Deep Feature Embedding

Chen Huang, Chen Change Loy, Xiaoou Tang

arXiv:1610.08904v1Oct 27, 2016cs.CVcs.LG

TL;DR

Global Euclidean metrics can misidentify hard samples when feature distributions are heterogeneous. The paper introduces an end-to-end PDDM unit that adapts similarity to local feature structure, yielding stronger retrieval, faster convergence, and improved open-set generalization. Its supported scope includes challenging retrieval datasets and transfer or zero-shot evaluation on ImageNet datasets.

Problem
Global Euclidean distance may not faithfully represent similarity in heterogeneous feature spaces, undermining hard-sample mining.
Method
PDDM learns a local-adaptive similarity metric from feature differences and feature means, and jointly trains metric and embedding objectives end-to-end.
Results
The method outperforms state-of-the-art retrieval baselines, converges faster, and generalizes to new classes in transfer and zero-shot learning.
Takeaways & Limitations
Local similarity-aware embedding supports high-quality online hard-sample mining and large-margin representations for retrieval and open-set recognition.
Takeaways & Limitations
The Fisher-like alternative assumes Gaussian class distributions and incurs O(m^2) mini-batch cost, an assumption and cost not suited to the paper’s heterogeneous-data setting.

Abstract

from arXiv · show

Existing deep embedding methods in vision tasks are capable of learning a compact Euclidean space from images, where Euclidean distances correspond to a similarity metric. To make learning more effective and efficient, hard sample mining is usually employed, with samples identified through computing the Euclidean feature distance. However, the global Euclidean distance cannot faithfully characterize the true feature similarity in a complex visual feature space, where the intraclass distance in a high-density region may be larger than the interclass distance in low-density regions. In this paper, we introduce a Position-Dependent Deep Metric (PDDM) unit, which is capable of learning a similarity metric adaptive to local feature structure. The metric can be used to select genuinely hard samples in a local neighborhood to guide the deep embedding learning in an online and robust manner. The new layer is appealing in that it is pluggable to any convolutional networks and is trained end-to-end. Our local similarity-aware feature embedding not only demonstrates faster convergence and boosted performance on two complex image retrieval datasets, its large margin nature also leads to superior generalization results under the large and open set scenarios of transfer learning and zero-shot learning on ImageNet 2010 and ImageNet-10K datasets.

1 Introduction

Deep embedding methods rely on global Euclidean distances for hard-sample mining, but heterogeneous feature distributions can make those distances misleading. The proposed PDDM unit adapts similarity to local feature structure and supports end-to-end embedding learning with improved retrieval efficiency and generalization.

Motivation: Hard sample mining is essential because easy examples often produce nearly zero loss, slowing convergence and weakening parameter updates.Existing methods commonly select hard negatives online using Euclidean distances in the embedding space.
Motivation: Global Euclidean distances can make intraclass pairs appear farther apart than interclass pairs in complex visual feature spaces.This overlap can mislead hard-sample selection and subsequent embedding learning.
Proposed approach: PDDM learns a locally adaptive metric from absolute feature differences and feature means, which encode relative and absolute position.The unit is designed to jointly support metric learning, hard-sample selection, and embedding learning.
Proposed approach: PDDM is pluggable into existing convolutional networks and trained end-to-end within the deep embedding pipeline.The overall architecture uses shared CNN architectures and parameters across inputs.
Results: Experiments on CUB-200-2011 and CARS196 show stronger retrieval performance, lower computational cost, faster convergence, and generalization to new classes.The generalization evaluation includes transfer learning and zero-shot learning on ImageNet 2010 and ImageNet-10K.

2 Related work

Related work uses Euclidean or parametric Mahalanobis metrics to guide hard-sample mining and similarity learning. These global metrics are limited when feature distributions are heterogeneous, motivating a locally adaptive alternative.

Hard sample mining: Deep embedding methods commonly select hard samples in a mini-batch using Euclidean distances or distance-based relevance scores.Examples include semi-hard negative mining and online reservoir importance sampling for triplets.
Metric learning: Similarity metrics underpin hard-sample mining, with prior work encoding relationships through pairwise, triplet, or higher-order constraints.Higher-order methods can lift mini-batch pairwise distances into a dense distance matrix.
Metric learning: Euclidean and Mahalanobis metrics share a global formulation that is poorly suited to heterogeneous feature distributions.The paper therefore proposes a locally adaptive metric for hard-sample mining.

3 Local similarity-aware deep embedding

The method jointly learns a position-dependent similarity metric and deep embedding so local structure guides online hard-sample selection and feature optimization. Its double-header hinge objectives separate similarity distributions while retaining efficient, end-to-end training.

Online hard-sample mining: The network computes mini-batch similarity scores, selects one hard quadruplet from local positive and negative pairs, and feeds its samples through shared CNNs.The quadruplet contains the most dissimilar positive pair and associated hard negatives selected by the learned metric.
Position-dependent metric: PDDM adapts similarity scoring to absolute feature position, addressing heterogeneous embedding spaces where Euclidean and Mahalanobis metrics can select inappropriate hard samples.The unit uses relative and absolute feature positions rather than feature differences alone, defining a flexible nonlinear similarity function learned with the embedding.
Metric optimization: The metric loss locally maximizes the margin between the selected positive similarity and two negative similarities using slack variables and enforced margin α.The local inter-distribution margin is approximated by min(S_i,j − S_i,k, S_i,j − S_j,l), followed by a double-header hinge loss.
Joint metric and embedding optimization: The joint objective combines metric loss, embedding loss, and regularization, providing supervision at both score and feature levels.Feature-level supervision constrains Euclidean distances within the PDDM-selected quadruplet, while score-level supervision learns the metric and embedding together.
Comparison with existing embeddings: Unlike contrastive, triplet, and lifted structured embeddings, the proposed method uses genuinely hard local examples to avoid updating negative pairs toward the wrong gradient direction.Figure 3 illustrates this distinction for a class whose local structure lacks a safe global margin.

4 Results

Experiments on disjoint-class fine-grained retrieval and ImageNet transfer settings evaluate PDDM's local similarity metric, embedding quality, convergence, and generalization. PDDM improves retrieval learning and supports strong performance on new classes.

Image retrieval: Experiments use disjoint training and testing classes from CUB-200-2011 and CARS196, with Recall@K as the retrieval metric.CUB-200-2011 uses 100 classes for training and 100 for testing; CARS196 uses 98 classes for each split.
Convergence: 2× faster convergence and lower converged loss are achieved than regular Euclidean hard quadruplet mining over 200 epochs.Both approaches have near-linear mini-batch complexity, unlike the quadratic complexity of [29].
Retrieval behavior: PDDM handles large intraclass variation by producing stable similarity scores for differently scattered features around a query and generalizes across disjoint class labels.Figure 4 reports the retrieved images, similarity scores, feature distances, and corresponding embeddings.
Retrieval performance: PDDM+Quadruplet yields 8%+ Recall@K=1 gains over previous Euclidean hard-mining methods, while PDDM also improves triplet and quadruplet embeddings.The learned features are described as well-clustered, with sharp boundaries and large margins between many classes.
Ablation and settings: Removing embedding-loss supervision reduces Recall@K=1 by 3.4%+ for PDDM score retrieval and 6.5%+ for PDDM+Quadruplet.A batch size below 64 also reduces CUB-200-2011 Recall@K=1 to 55.7%, whereas m=132 gives marginal gains with substantially longer training.
Transfer and zero-shot learning: PDDM features outperform state-of-the-art methods on ImageNet-10K transfer learning and achieve top results among competing deep CNN methods for ImageNet 2010 zero-shot learning.Transfer learning uses flat top-1 accuracy; zero-shot learning uses flat top-5 accuracy.

5 Conclusion

The paper presents end-to-end local similarity-aware embedding through PDDM and a double-header hinge loss. Experiments support its value for hard-sample mining, image retrieval, and generalization to new classes.

Contribution: PDDM adaptively measures local similarity in heterogeneous feature spaces to support high-quality online hard-sample mining.The method learns local similarity-aware deep feature embeddings end-to-end.
Optimization: A double-header hinge loss jointly optimizes the similarity metric and feature embedding under a large-margin criterion.
Conclusion: Experiments demonstrate efficacy on challenging image retrieval tasks and potential generalization to new classes in transfer-learning and zero-shot-learning settings.The paper identifies shared attributes or visual-semantic embeddings as future directions for zero-shot learning.