PaperlayerView exact revision on arXiv

Source-linked AI summary

Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval

Andrew Brown, Weidi Xie, Vicky Kalogeiton, Andrew Zisserman

arXiv:2007.12163v2cs.CV

TL;DR

Ranking-based metrics such as AP are difficult to optimize directly because their discrete ranking functions are non-differentiable. The paper introduces Smooth-AP, a sigmoid-smoothed AP objective for end-to-end deep-network training, and reports improvements across standard and large-scale retrieval datasets. The results support Smooth-AP as an effective and scalable plug-and-play approach for image retrieval.

  • Problem

    Average Precision is difficult to optimize directly with gradient descent because its ranking function is discrete, non-differentiable, and non-decomposable.

  • Method

    Smooth-AP replaces AP’s Indicator function with a temperature-controlled sigmoid, forming a differentiable plug-and-play objective for end-to-end deep-network training.

  • Results

    Smooth-AP outperforms recent AP-approximation and deep metric-learning methods across SOP, VehicleID, INaturalist, VGGFace2, and IJB-C, with especially large gains on larger-scale datasets.

  • Takeaways & Limitations

    Directly optimizing a smoothed AP objective preserves the ranking goal rather than optimizing embedding distances, while remaining simple and scalable to large retrieval datasets.

Abstract

from arXiv · show

Optimising a ranking-based metric, such as Average Precision (AP), is notoriously challenging due to the fact that it is non-differentiable, and hence cannot be optimised directly using gradient-descent methods. To this end, we introduce an objective that optimises instead a smoothed approximation of AP, coined Smooth-AP. Smooth-AP is a plug-and-play objective function that allows for end-to-end training of deep networks with a simple and elegant implementation. We also present an analysis for why directly optimising the ranking based metric of AP offers benefits over other deep metric learning losses. We apply Smooth-AP to standard retrieval benchmarks: Stanford Online products and VehicleID, and also evaluate on larger-scale datasets: INaturalist for fine-grained category retrieval, and VGGFace2 and IJB-C for face retrieval. In all cases, we improve the performance over the state-of-the-art, especially for larger-scale datasets, thus demonstrating the effectiveness and scalability of Smooth-AP to real-world scenarios.

1 Introduction

Image retrieval ranks all images by relevance to a query, but directly optimizing AP is difficult because ranking is discrete, non-differentiable, and non-decomposable. Smooth-AP addresses this challenge with a differentiable AP approximation for end-to-end deep-network training and reports strong retrieval results.

  • Motivation: Query-by-example retrieval ranks all instances by relevance, prioritizing high recall when users seek all matching images rather than one sufficient hit.The paper contrasts searching a personal photo collection or licensing site with object identification applications where one match is enough.
  • Motivation: Average Precision is a benchmark retrieval metric whose discrete ranking function makes direct gradient-based optimization notoriously difficult.The ranking function is also non-decomposable, preventing straightforward end-to-end optimization.
  • Contribution: Smooth-AP replaces the non-differentiable AP objective with a differentiable approximation, enabling end-to-end training of deep networks for ranking-based tasks.The method is presented as a plug-and-play objective that relaxes the Indicator function with a sigmoid.
  • Related work: Metric-learning losses use pairs, triplets, or lists, but limited rank awareness or distance-focused optimization can neglect the importance of correcting high-ranked ordering errors.These approaches may spend capacity improving low-ranked positives while remaining less aligned with rank-based evaluation.
  • Related work: Direct AP-optimization research addresses non-differentiability and non-decomposability through ranking distributions, loss-augmented inference, direct loss minimization, and smooth bounds.The paper positions Smooth-AP within this broader family of AP-approximation approaches.

3 Background

The paper defines retrieval rankings, positive and negative sets, AP, and the Indicator-based ranking function used to compute it. Because the Indicator is flat or discontinuous, its derivative prevents gradient-based optimization.

  • Task definition: A retrieval system ranks every instance in a set Ω by relevance to a query, partitioning instances into query-specific positive and negative sets.Positive instances share the query’s class, while negative instances belong to different classes.
  • Average Precision: Average Precision is the area under a precision-recall curve computed from predicted relevance scores for a query.The paper uses cosine similarity for relevance scoring, although Smooth-AP is independent of that choice.
  • Average Precision: AP compares each positive instance’s rank within the positive set with its rank within the full retrieval set.The paper assumes proper rankings in which no two samples are tied.
  • Ranking function: The ranking function uses an Indicator function to count instances according to their relevance-score differences, and it can be implemented with a difference matrix D ∈ R^m×m.This ranking construction supplies the discrete component used in the AP computation.
  • Indicator derivatives: The AP Indicator is a Heaviside step function whose distributional derivative is a Dirac delta, leaving it flat or discontinuous for gradient-based optimization.These derivative properties motivate replacing the Indicator with a smooth approximation.

4 Approximating Average Precision (AP)

Smooth-AP makes Average Precision directly optimizable by replacing AP’s non-differentiable indicator with a temperature-controlled sigmoid. The formulation emphasizes ranking errors according to their impact on AP rather than treating all distance violations equally.

  • 4.1 Smoothing AP: Smooth-AP replaces AP’s discrete indicator function with a sigmoid, creating a differentiable objective for end-to-end network training.The sigmoid temperature τ controls the sharpness of the approximation.
  • 4.1 Smoothing AP: As τ approaches 0, the smoothed approximation becomes tighter and converges toward the indicator function.
  • 4.1 Smoothing AP: The smoothing parameter τ defines the operating region where mis-ranked score differences receive gradients and are shifted toward the correct order.Smaller τ gives a smaller gradient region and a tighter approximation to true AP.
  • 4.1 Smoothing AP: AP optimization targets violating positive-negative pairs, where a negative instance ranks above a positive instance; optimal AP places all positives above all negatives.
  • 4.1 Smoothing AP: Triplet loss treats pairwise terms equally and optimizes distances linearly, although correcting high-rank ordering errors can affect AP more than reducing an already-correct distance.The paper illustrates this with violations such as s4−s1 versus s7−s4 and distance changes that do not shift ranking order.
  • 4.1 Smoothing AP: Compared with FastAP and Blackbox AP, Smooth-AP is presented as a closer approximation to AP and simpler to implement because it only replaces the indicator with a sigmoid.FastAP uses histogram binning, whereas Blackbox AP uses a variant of numerical derivative.

5 Experimental Setup

The experiments fine-tune pretrained networks with Smooth-AP across five retrieval datasets spanning products, vehicles, species, and faces. Evaluation uses retrieval protocols based on mAP and Recall@K, with dataset-specific backbones and preprocessing.

  • 5 Experimental Setup: The evaluation covers five datasets, including standard retrieval benchmarks and additional large-scale datasets with more than 100K images.
  • 5.1 Datasets: VehicleID contains 221,736 images across 26,267 vehicle categories, with small, medium, and large test sets containing 800, 1,600, and 2,400 classes.
  • 5.1 Datasets: INaturalist contains 461,939 images from 8,142 species classes, with 5,690 classes for training and 2,452 unseen classes for retrieval testing.The task is designed around visually similar species captured in varied environments.
  • 5.1 Datasets: VGGFace2 provides over 3.31 million images from 9,131 subjects, while its test split contains 500 identities and 169K images.
  • 5 Experimental Setup: For every dataset, each class instance serves in turn as a query against the remaining instances, and performance is evaluated using mAP and Recall@K.Object and animal retrieval uses Recall@K to compare with existing work.
  • 5.2 Implementation Details: Object and animal retrieval uses ImageNet-pretrained ResNet-50 networks, while face retrieval uses SENet-50 and ArcFace models trained on VGGFace2.

6 Results

Smooth-AP is evaluated across standard and large-scale image and face retrieval benchmarks, where it consistently achieves strong retrieval performance. Ablations further examine sigmoid temperature, positive-set size, and batch size, while the authors relate gains to tighter AP approximation and scalability.

  • Stanford Online Products: Smooth-AP achieves state-of-the-art results on Stanford Online Products, outperforming Blackbox AP and FastAP by 1.5% for Recall@1.It performs on par with Cross-Batch Memory while using only a single mini-batch per training iteration.
  • VehicleID and INaturalist: Smooth-AP achieves state-of-the-art VehicleID results, outperforming FastAP by 3% for Small-protocol Recall@1 and exceeding Cross-Batch Memory on 4 of 6 recall metrics.
  • VehicleID and INaturalist: On INaturalist, Smooth-AP outperforms re-trained metric-learning and AP-approximation methods by 2–5% on Recall@1 at matched batch size.Increasing the batch size from 224 to 384 further raises Recall@1 to 66.6, a 1.4% improvement.
  • Face Retrieval: Appending Smooth-AP improves face-retrieval mAP by up to 4.4% on VGGFace2 and 3.1% on ArcFace, while outperforming pairwise and triplet losses.The evaluation uses strong face-verification baselines on VGGFace2 and IJB-C.
  • Ablation Study: A sigmoid temperature of 0.01 provides the best mAP trade-off between AP-approximation tightness and a sufficiently large gradient operating region.Lower temperature produces a tighter approximation, but the operating region becomes smaller.
  • Ablation Study: Positive-set size 4 produces the highest mAP, while larger batch sizes improve mAP by increasing the chance of sampling hard negatives.The positive-set effect is attributed to class-level mini-batch sampling and harder ranking violations.
  • Further Discussion: Across SOP, VehicleID, INaturalist, VGGFace2, and IJB-C, the authors attribute Smooth-AP’s gains to tighter AP approximation and report scalability to large retrieval datasets.They also note that AP formulation naturally incorporates properties often handcrafted in metric-learning losses.

7 Conclusions

Smooth-AP directly optimizes a smoothed approximation of AP, outperforming prior AP-optimizing and deep metric-learning losses while scaling to large-scale retrieval.

  • Smooth-AP directly optimizes a smoothed approximation of Average Precision rather than using a surrogate to encourage ranking.
  • The method preserves AP's ranking objective instead of optimizing distances in the embedding space.
  • Smooth-AP outperforms recent AP-optimizing methods and deep metric-learning methods.
  • Fine-tuning face-verification networks with Smooth-AP strongly improves performance.
  • Experiments on several large-scale datasets show that Smooth-AP is considerably more scalable than previous approximations.

Supplementary Material

The supplementary material identifies the Visual Geometry Group at the University of Oxford and provides an associated contact address and research webpage.

  • The work is associated with the Visual Geometry Group at the University of Oxford.
  • The supplementary material lists the contact address {abrown,weidi,vicky,az}@robots.ox.ac.uk.
  • A Smooth-AP research webpage is provided at the University of Oxford domain.

1 Further qualitative results

Qualitative retrieval examples compare baseline models with models trained after appending Smooth-AP across VGGFace2, online products, and INaturalist. The examples report improved AP and fewer false positives after adding Smooth-AP.

  • Qualitative displays mark the query in blue, same-class retrieved instances in green, and other retrieved instances in red.
  • VGGFace2 Test set: 52.9% Average Precision improvement is shown for a VGGFace2 query after appending Smooth-AP to the SENet-50 baseline.
  • VGGFace2 Test set: 54.9% Average Precision improvement is shown for another VGGFace2 query, with all false positives removed from the top-ranked results.
  • VGGFace2 Test set: 41.8% Average Precision improvement and removal of all false positives are reported for a third VGGFace2 query.
  • VGGFace2 Test set: 24.8% Average Precision improvement is reported when Smooth-AP is appended to the ArcFace ResNet-50 baseline, again removing all top-ranked false positives.
  • Stanford Online Products: The online-products examples use small positive sets, with |P| = 11, 5, 7, 11 for the four queries.
  • INaturalist: INaturalist examples show baseline false positives and large Average Precision gains after appending Smooth-AP.

2 Source code

The source code implements Smooth-AP as a short PyTorch-style function that transforms relevance-score differences with a temperature-controlled sigmoid and returns one minus approximate AP.

  • The paper presents the method as a short PyTorch-style pseudocode implementation.
  • A temperature-controlled sigmoid approximates the Heaviside indicator used to obtain differentiable rankings.
  • The implementation defines a smooth_ap function that accepts predicted relevance scores and ground-truth relevance scores.
  • The code repeats scores into a matrix, transposes it, and computes pairwise score differences for ranking.
  • The implementation computes positive-instance rankings, approximate AP, and returns 1-AP as the loss.

3 Details on the effects of increasing the mini-batch size on Smooth-AP

Larger mini-batches increase the proportion of Smooth-AP difference-matrix elements receiving non-negligible sigmoid gradients, helping explain its improved behavior with larger batches.

  • The increased likelihood of nearby relevance scores in larger mini-batches places more difference-matrix elements within the sigmoid’s narrow operating region.
  • Larger mini-batches increase P, the proportion of difference-matrix elements within the sigmoid’s operating region.P is averaged across mini-batches during one Online Products training epoch, using 0.005 to represent a non-negligible gradient.
  • More elements receiving non-negligible gradients explains why Smooth-AP benefits from larger mini-batch sizes.

4 Choice of hyper-parameters for the compared-to methods for the INaturalist experiments

For INaturalist comparisons, FastAP and Blackbox AP use hyper-parameters recommended for the largest datasets in their respective publications, while the authors note that tuning could improve Recall@K.

  • FastAP uses 20 histogram bins, while Blackbox AP uses λ=4 and margin=0.02.
  • The compared methods’ settings follow recommendations for the largest datasets evaluated in their respective publications.These datasets were selected as closest to INaturalist in training-image count.
  • Varying these hyper-parameters might increase the evaluated Recall@K scores.

5 Complexity of the proposed loss

Smooth-AP has a slightly higher time complexity than the compared AP-optimising losses, but this produces a negligible increase in training time when appended to ResNet50.

  • Table S1 compares time complexities and measures forward-backward times for the backbone and appended loss during one training iteration.
  • The timing measurements use a ResNet50 backbone, reporting separate backbone time and loss time averages.
  • Smooth-AP’s slightly increased time complexity leads to a negligible increase in training time.
Loading 2007.12163v2…