PaperlayerView exact revision on arXiv

Source-linked AI summary

Global Average Precision for Representation Learning

Bill Psomas, Mohammad Mahdi, Michalis Thomas, Danda Pani Paudel, Giorgos Tolias, Giorgos Kordopatis-Zilos

arXiv:2610.09863v1cs.CVcs.LG

TL;DR

Per-query metrics and losses do not ensure similarities are comparable across queries, a property needed when one threshold serves all queries. The paper introduces gSAP, a drop-in surrogate for global Average Precision, and reports improvements across representation-learning settings, including under a shared threshold.

  • Problem

    Per-query metrics and losses assess rankings within queries but leave cross-query similarity comparability unmeasured, although a single decision threshold depends on it.

    Source: §1 · pp. 1–2

  • Method

    gSAP is a smooth surrogate that ranks the full similarity matrix jointly, using the same similarity matrix and binary relevance labels as existing losses.

    Source: §1 · pp. 1–2

  • Results

    Across supervised, cross-modal, and self-supervised training, gSAP improves gAP; at 0.90 precision, it retrieves 4× as many matches as mSAP on iNaturalist.

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    More consistent similarities across queries yield gains when a single threshold serves all queries.

    Source: §1 · p. 2

    Its cost is quadratic in batch size, which negative dropping mitigates.

    Source: §5 · p. 9

Abstract

from arXiv · show

Standard information retrieval metrics, such as mean Average Precision (mAP), assess performance one query at a time, based on how the similarities between a query and its positives compare against those with its negatives. The same holds for common representation learning losses, such as InfoNCE and per-query AP surrogates. None of them considers whether similarities are comparable across queries, which any system with a single decision threshold relies on. Global Average Precision (gAP) does, by ranking all query-candidate pairs in one list and computing a single AP. We introduce gSAP, a differentiable surrogate of gAP. It needs only a similarity matrix and a binary matrix marking the positive pairs, the same input as existing losses, so it is a drop-in replacement for them and agnostic to the encoder, the modality, and the source of supervision. Since it considers all possible pairwise comparisons in the batch jointly, it also remains trainable at low temperatures, a regime where per-query surrogates run out of gradient. Swapping it into established recipes improves supervised metric learning, cross-modal alignment, and self-supervised pretraining, where, to our knowledge, it is the first ranking loss to replace the community standard InfoNCE in the latter two. Its similarities are more consistent across queries, which drives the gains under a universal threshold. gSAP retrieves up to four times as many positive pairs as the strongest AP surrogate at the same precision, and it degrades the least when queries with no positives in the database are added. Beyond thresholding, models trained with gSAP also learn better representations, with higher transfer, $k$NN and zero-shot classification accuracy.

1 INTRODUCTION

gAP evaluates ranking and similarity consistency across queries, addressing a limitation of per-query metrics. The paper introduces gSAP, a smooth, drop-in surrogate, and reports gains across supervised, cross-modal, and self-supervised training.

1 INTRODUCTION

  • Unlike mAP, gAP ranks all query–database pairs together, capturing similarity consistency across queries needed when systems use a single decision threshold.Per-query mAP can leave similarity distributions varying from query to query.Source: §1 · p. 1
  • gSAP is a smooth surrogate that directly optimizes gAP and replaces similarity-matrix-based losses; negative dropping helps scale it to large batches.The approach prunes negative similarities that have little effect on the loss.Source: §1 · pp. 1–2
  • Because each positive is compared with every batch similarity, gSAP retains unsaturated comparisons at low temperatures where per-query surrogates lose gradient.This permits a closer AP approximation without vanishing gradients.Source: §1 · p. 2
  • Using only a similarity matrix and binary relevance labels, gSAP applies across encoders, modalities, and supervision sources, and is the first ranking loss to replace InfoNCE in CLIP-style cross-modal and self-supervised training.Relevance can mark same-class pairs, cross-modal matches, or augmented views of the same sample.Source: §1 · p. 2
  • Across supervised, cross-modal, self-supervised, and distractor-query tasks, gSAP achieves the best gAP on every retrieval benchmark and yields better representations.Its cross-query consistency also drives gains when one threshold serves all queries.Source: §1 · p. 2

2 RELATED WORK

Prior representation-learning objectives mostly rank pairs within individual queries, leaving similarities across queries unconstrained. The paper distinguishes its batch-wide ranking objective from per-query surrogates and related batch-wise detection methods.

2 RELATED WORK

  • Most representation-learning losses operate per pair, query, or class proxy; InfoNCE normalizes rows and columns, and these objectives leave cross-query similarity comparisons unconstrained.Triplet and Multi-Similarity losses weight pairs relative to an anchor [Schroff et al., 2015; Wang et al., 2019b], while InfoNCE is standard in vision–language and self-supervised learning [Oord et al., 2018; Radford et al., 2021; Chen et al., 2021].Source: §2 · p. 3
  • Per-query AP and Recall@k surrogates depend on within-query orderings, while GlobalAP still averages AP over queries despite its name.Per-query AP surrogates include Rolínek et al. (2020), Revaud et al. (2019), Brown et al. (2020), and Ramzi et al. (2021); Recall@k is addressed by Patel et al. (2022).Source: §2 · p. 3
  • Unlike batch-wise AP optimization for object detection, this approach directly ranks all batch samples with one surrogate and requires neither extra regularization nor post-hoc operations.Object-detection work optimizes AP across anchors and classes using gradient approximation [Chen et al., 2019].Source: §2 · p. 3

3 METHOD

The method replaces per-query ranking with a global ranking objective and smooths AP for differentiable training. Its batch analysis finds that gSAP maintains useful gradients at low temperatures where mSAP stalls.

3.1 AVERAGE PRECISION

  • AP summarizes a ranked list through precision at positive-item ranks, while mAP averages per-query AP and gAP ranks all query–database pairs jointly.Unlike mAP, gAP reflects both per-query ranking quality and similarity consistency across queries.Source: §3.1 · p. 3, §3.2 · p. 4

3.5 ANALYSIS BY SINGLE BATCH OVERFITTING

  • AP is optimal when all positives rank ahead of all negatives.The single-batch experiment uses the same initialization and batch for 20 updates across four temperatures.Source: §3.1 · p. 3, §3.5 · p. 5

3.3 SURROGATE LOSSES

  • Replacing the discrete Heaviside indicator with a temperature-controlled sigmoid makes AP differentiable; gSAP applies smoothed AP to the flattened similarity matrix.Smaller temperatures better approximate the exact metric but saturate comparisons sooner, hindering training.Source: §3.3 · p. 4

3.4 TRAINING WITH GLOBAL AVERAGE PRECISION

  • Training uses batch similarity and binary relevance matrices; negative dropping removes the α fraction of smallest negatives while keeping positives.Dropping up to 90% alters only the low-precision tail, trading fidelity for lower memory use and potentially larger batches.Source: §3.4 · p. 5, Figure 2 · p. 5

3.5 ANALYSIS BY SINGLE BATCH OVERFITTING

  • At τ = 10^-4, gSAP retains more than 150 unsaturated comparisons per positive and reaches the other temperatures’ mAP and gAP, while mSAP has fewer than one and stalls.At τ = 0.1, both surrogates are poor metric proxies and finish below the other settings.Source: Figure 3 · p. 6, §3.5 · p. 6

4 EXPERIMENTS

Across retrieval, classification, self-supervised learning, metric learning, and distractor-query tasks, gSAP improves global ranking performance while remaining competitive on per-query metrics. Its advantages are clearest when one threshold must serve all queries.

Cross-modal retrieval

  • Across six image and video datasets, gSAP leads InfoNCE and SigLIP in every reported retrieval metric and direction.The comparisons cover gAP, mAP, and Recall@5 on image datasets and gAP and Recall@k on video datasets.Source: §4 · p. 7, Table 1 · p. 7, Table 2 · p. 7

Zero-shot classification

  • Across seven image and video datasets, gSAP outperforms InfoNCE and SigLIP in every reported zero-shot classification metric.Across 108 scores on the full 40-dataset benchmark, gSAP improves on average by +0.49 over InfoNCE and +0.64 over SigLIP.Source: §4 · p. 7, Table 3 · p. 7

Self-supervised learning

  • In self-supervised learning, gSAP trails only on in-distribution linear probing at ViT-B, with consistently larger gains under kNN.Transfer gains are concentrated on six fine-grained datasets; the same substitution also improves music tagging in audio contrastive learning.Source: §4 · pp. 7–8, Table 4 · p. 7

Metric learning

  • On SOP and iNaturalist, gSAP achieves the best gAP by a clear margin while keeping mAP and R@1 close to the best.Ranking-loss competitors match gSAP on mAP and R@1, while their lower gAP reflects less consistent similarity distributions across queries.Source: §4 · p. 8, Table 5 · p. 8

Consistency and thresholding

  • With a shared threshold at 0.90 precision, gSAP retrieves 12% more matches than mSAP on SOP and 4× as many on iNaturalist.Its per-query optimal thresholds are more concentrated around the shared threshold, reducing the gap between per-query and global F1.Source: §4 · p. 8, Table 6 · p. 8, Figure 4 · p. 8

Comparison with state-of-the-art methods

  • With negative dropping at a 95% rate and batch size 4,000, gSAP ranks among state-of-the-art methods across two datasets and two backbones.The reported comparison uses ResNet50 and ViT-B/16.Source: §4 · p. 9, Table 7 · p. 9

Tasks with distractor queries

  • On four distractor-query tasks, gSAP attains the best gAP and the smallest drop before score normalization.On Met, gSAP is 4.5 gAP ahead of mSAP when distractors are included; normalization brings losses' drops closer but does not close the DISC21 gAP gap.Source: §4 · p. 9, Table 8 · p. 9

5 CONCLUSIONS

The paper proposes gSAP as a differentiable surrogate for global AP and summarizes its relation to query-centric metrics and losses. The related work spans metric learning, multimodal alignment, self-supervision, ranking surrogates, calibration, and retrieval metrics.

5 CONCLUSIONS

  • gSAP directly optimizes gAP, improving supervised metric learning, cross-modal alignment, and self-supervised pretraining by replacing existing losses without other recipe changes.Its quadratic batch cost is mitigated by negative dropping, while some achievable similarity consistency remains unrealized.Source: §5 · p. 9

A EXTENDED RELATED WORK

  • Unlike pair-, anchor-, or proxy-based metric-learning losses, gSAP couples all batch pairs through one ranking.Prior approaches include pairwise and triplet losses, listwise and N-pair losses, proxy methods, and angular-margin losses.Source: A EXTENDED RELATED WORK · p. 18
  • Prior vision–language objectives use row-normalized contrastive, pairwise sigmoid, or listwise ranking losses; this work targets one AP over all image–text or audio–text pairs.SigLIP scores pairs independently at a shared operating point, whereas the proposed objective targets gAP directly.Source: A EXTENDED RELATED WORK · p. 18
  • gSAP replaces InfoNCE in contrastive self-supervision because augmented views define relevance, but it does not apply to distillation or reconstruction losses lacking ranked negatives.The cited non-contrastive approaches include teacher–student distillation and masked image modeling.Source: A EXTENDED RELATED WORK · p. 19
  • Existing differentiable ranking surrogates optimize query-level AP or other ranking metrics; they do not target global criteria such as gAP.Examples include Smooth-AP, FastAP, ROADMAP, and GlobalAP, which still averages AP over queries.Source: A EXTENDED RELATED WORK · p. 19
  • Cross-query similarity consistency has been pursued with training objectives, regularizers, and post-hoc mappings; gSAP needs neither an auxiliary regularizer nor post-hoc mapping.The paper states that adding a consistency regularizer on top brings no further gain.Source: A EXTENDED RELATED WORK · p. 19
  • Query-centric mAP and Recall@k differ from gAP, which aggregates pairs into one precision–recall curve under a shared similarity threshold.The paper adopts gAP as a primary metric and derives a surrogate loss to optimize it directly.Source: A EXTENDED RELATED WORK · p. 19

MS COCO

  • The MS COCO appendix figure compares zero-shot retrieval gAP across batch sizes for gSAP, InfoNCE, and SigLIP after CC12M fine-tuning.The models fine-tune pretrained OpenAI CLIP ViT-B/32 for five epochs.Source: MS COCO · p. 20

B.1 CROSS-MODAL LEARNING

Across vision–language settings, gSAP is the best or tied-best loss in retrieval and classification, and it also improves audio–text retrieval and audio classification. Its gains persist with smaller batches, negative dropping, other backbones, and composed image retrieval.

Audio–language.

  • gSAP outperforms CLIP and SigLIP on every reported audio retrieval and classification column, with its clearest retrieval margin on Clotho evaluation.SigLIP falls below the pretrained model on ESC-50.Source: §B.1 · p. 20

Batch size sensitivity.

  • At every tested batch size, gSAP achieves the highest gAP on Flickr8k, Flickr30k, and MS COCO, with its largest margin at the smallest batch.At batch size 512, it exceeds InfoNCE trained with four times the batch size on Flickr30k and MS COCO.Source: §B.1 · p. 20

Negative dropping.

  • Dropping 95% of negatives costs at most 0.9 gAP; increasing the resulting batch from 2,304 to 4,096 gives gSAP its best result in every metric.Even at batch size 2,304, gSAP remains ahead of InfoNCE and SigLIP in every metric on all three image datasets.Source: §B.1 · pp. 20–21

Vision–language.

  • gSAP is the best or tied-best loss across image and video retrieval, composed image retrieval, and zero-shot classification with alternative backbones and pretraining data.On MSR-VTT with ViT-B/16, it beats SigLIP by 3.3 R@1; with LAION-400M, it loses the least when no loss recovers pretrained video→text recall.Source: §B.1 · p. 22

Zero-shot composed image retrieval and classification.

  • For both alternative backbones, gSAP is best in every metric on ConCon-Chi, while its zero-shot classification is best or tied-best across all seven datasets.On MS COCO with ViT-B/16, it doubles the pretrained model's gAP.Source: §B.1 · p. 22

B.2 SELF-SUPERVISED LEARNING

With self-supervised pretraining, gSAP improves transfer especially on fine-grained datasets, leads or ties on distribution-shift and corruption evaluations, and also improves audio music tagging. Its advantage is less pronounced on coarse-grained transfer datasets.

In- and out-of-distribution performance.

  • For ViT-S, gSAP leads under both probes on every distribution-shift set; for ViT-B, it leads on every set under kNN and ties InfoNCE under linear probing overall.Neither loss trades in-distribution accuracy for robustness: the OOD ordering follows the in-distribution ordering per probe.Source: §B.2 · pp. 22–23

Transfer learning.

  • On six fine-grained transfer datasets, gSAP is clearly ahead under both linear probing and kNN at both model scales, while results on four coarse-grained sets are close.InfoNCE is slightly ahead at ViT-B on the coarse-grained sets.Source: §B.2 · pp. 22–23

Robustness to corruptions.

  • On ImageNet-C, gSAP leads under both probes with ViT-S; with ViT-B, it ties under linear probing and leads under kNN across families and severities.Its margin under kNN shrinks as corruptions grow stronger, while it retains a slightly larger share of clean accuracy under both probes.Source: §B.2 · p. 22, Table I · p. 23

Audio-based SSL.

  • Replacing InfoNCE with gSAP improves both ROC-AUC and PR-AUC in CLMR music tagging, with the larger gain in PR-AUC.PR-AUC is the tagging analogue of gAP, and gSAP directly optimizes a closely related objective.Source: §B.2 · p. 23

B.3 METRIC LEARNING

In limited-data settings, gSAP consistently achieves the strongest gAP, while its performance plateaus at low temperatures and benefits from larger batches. Additional analyses find that its gains are not specific to one AP approximation and that similarity-scale regularizers add little to gSAP.

Comparison on limited data settings.

  • At ρ=2, gSAP reaches up to 6 gAP above the second-best method and often beats competitors trained with more than twice the data when γ is small.At ρ=4, it remains best; all three losses have nearly identical mAP, and trained models generally surpass the pretrained model.Source: §B.3 · p. 24

Impact of hyper-parameters.

  • gAP peaks and plateaus for τ below 0.01, while larger τ values above 0.1 yield very low performance.Among tested drop rates, the larger batch size that fits in memory gives better performance.Source: §B.3 · p. 24

Combination with similarity-scale regularizers.

  • Adding TCM helps per-query losses generally but inconsistently, adds nothing on top of gSAP, and KoLeo hurts gAP everywhere.TCM constrains similarity scales across queries, which gSAP already achieves through its loss alone; the explicit regularizer adds hyperparameters without consistent effectiveness.Source: §B.3 · p. 25

Performance with other AP approximations.

  • Across Smooth-AP, SoftBin, BlackBox, and ROADMAP, each gAP surrogate beats its mAP counterpart in gAP at similar mAP and R@1.This pattern attributes the gain to ranking the whole similarity matrix rather than to a particular AP approximation.Source: §B.3 · p. 25, Table L · p. 25

Impact of negative dropping.

  • Dropping up to 99.9% of negatives has only a marginal impact, but performance degrades when very few negatives remain.At α=99%, increasing batch size improves gAP even with most easy negatives discarded.Source: §B.3 · p. 25

B.4 TASKS WITH DISTRACTOR QUERIES

Across distractor-query tasks, gSAP improves global-threshold outcomes and retains strong per-query performance; its advantage persists across backbones, while score normalization helps only in some settings. The appendix also documents cross-modal and self-supervised training and evaluation protocols.

B.4 TASKS WITH DISTRACTOR QUERIES

  • For gSAP on SOP, negative dropping permits larger batches on the same GPU, which improves gAP.The analysis varies the fraction of lowest negative similarities dropped.Source: Table M · p. 25
  • On benchmark-specific per-query metrics, all fine-tuned losses improve over baseline on every column, and gSAP is best or tied-best on every column.The fine-tuned losses differ much less on these metrics than on gAP.Source: §B.4 · pp. 25–26
  • At a shared threshold, gSAP retrieves the largest share of DISC21 copies at 90% precision and rejects the largest share of unknown WildlifeReID-10k queries at the specified operating point.On WildlifeReID-10k, the two AP surrogates are on par for known queries at the same threshold.Source: §B.4 · p. 26
  • Across ILIAS backbones, gSAP has the best gAP with and without distractors and the best mAP; adding distractor queries sharply lowers gAP for every loss and backbone.Score normalization raises gAP for two backbones without distractors, but never raises gAP with distractors.Source: §B.4 · pp. 26–27

C.1 CROSS-MODAL LEARNING

  • Cross-modal experiments fine-tune image–text and audio–text models, then evaluate zero-shot retrieval and classification across image, video, and audio benchmarks.The vision–language protocol uses OpenCLIP and CC12M fine-tuning; the audio–language protocol fine-tunes LAION-CLAP on AudioCaps.Source: §C.1 · pp. 27–28

C.2 SELF-SUPERVISED LEARNING

  • Self-supervised experiments replace InfoNCE with gSAP in the unchanged MoCo v3 and CLMR recipes, evaluating frozen-feature classification or tagging.MoCo v3 uses ImageNet-1k pretraining; CLMR uses MagnaTagATune audio pretraining.Source: §C.2 · pp. 28–29

C.3 METRIC LEARNING

The metric-learning experiments use SOP and iNaturalist with prior-work splits, and evaluate models under specified tuning, training, and threshold-selection protocols. A scarce-data protocol also varies the amount of training data retained.

C.3 METRIC LEARNING

  • SOP uses all test images as queries, while iNaturalist samples four images per test class, yielding 9,808 queries for mAP and gAP.The training and test splits contain 59,551 and 60,502 images for SOP, and 325,846 and 136,093 for iNaturalist, respectively.Source: §C.3 · p. 29
  • Hyperparameters are selected by 10 Optuna trials on a 70/30 training-validation split, then the final model is trained on the full training set and evaluated across three seeds.Learning rate and temperature are searched over 10^-5 to 5·10^-4 and 10^-4 to 5·10^-2, respectively.Source: §C.3 · p. 29
  • The threshold protocol fits thresholds on half the test classes and evaluates them on the other half, reporting F1g, R@P90, and per-query F1q.The scarce-data protocol also retains γ ∈ {1, 5, 10, 25, 50, 100}% of eligible training classes and draws ρ ∈ {2, 4} images per class, using three seeds per setting.Source: §C.3 · p. 30

C.4 TASKS WITH DISTRACTOR QUERIES

The distractor-query experiments cover retrieval and recognition tasks where some queries have no matching database item. Each task retains its official training and evaluation code, with only the loss replaced.

C.4 TASKS WITH DISTRACTOR QUERIES

  • DISC21, Met, ILIAS, and WildlifeReID-10k cover image copy detection, artwork recognition, large-scale instance retrieval, and individual-animal re-identification, respectively.Their queries include cases without a source, exhibit, or individual image in the database; ILIAS distractors are simulated.Source: §C.4 · p. 30
  • The experiments compare against each benchmark's stated reference model, averaging results over three seeds with shared training recipes for gSAP, InfoNCE, and mSAP.Unless validation tuning is specified, gSAP uses τ = 0.001, negative dropping α = 90%, and batches of 4,096 samples.Source: §C.4 · p. 30
  • gAP is measured both with distractor queries included and after their removal, while per-query metrics ignore distractors; DISC21 also reports recall at a global threshold giving 90% precision.For ILIAS, distractors are simulated using YFCC100M images, and chunked queries are evaluated against positives and distractors.Source: §C.4 · p. 31

D DETAILED RESULTS

Across two CLIP backbones, gSAP fine-tuning yields the strongest overall zero-shot classification performance, with higher averages than pretrained CLIP, CLIP InfoNCE, and SigLIP. Gains are broad, though isolated metrics sometimes favor other models.

D DETAILED RESULTS

  • 1.16, 0.49, and 0.64 points: gSAP improves over pretrained, CLIP InfoNCE, and SigLIP, respectively, across 108 classification scores with ViT-B/32.For ViT-B/16, the corresponding improvements are 1.43, 0.65, and 0.77 points.Source: D DETAILED RESULTS · p. 31
  • gSAP fine-tuning yields the strongest overall performance on both ViT-B/32 and ViT-B/16 across the full CLIP benchmark.The benchmark reports top-1, top-5, and mean per-class recall across 38 classification tasks; Tables R and S give per-dataset results.Source: D DETAILED RESULTS · pp. 31–33
  • Where gSAP is not best, pretrained models remain competitive on some tasks, while CLIP or SigLIP occasionally lead on specific metrics by typically small differences.Clear-margin underperformance by gSAP is rare; structured dSprites and SmallNORB remain near chance for all models.Source: D DETAILED RESULTS · p. 31

E THEORETICAL ANALYSIS

The analysis separates per-query ranking quality from cross-query similarity consistency: mAP and Recall@k are invariant to independent monotone transformations of query rows, but gAP is not. It defines a consistency ceiling and shows that gSAP raises efficiency relative to mSAP without reducing that ceiling.

E THEORETICAL ANALYSIS

  • Independent strictly increasing transformations of query rows preserve within-row rankings, so mAP and Recall@k stay fixed while gAP can change with the global interleaving.Thus gAP reflects both per-query rankings and consistency across queries.Source: E THEORETICAL ANALYSIS · pp. 34–35
  • gSAP's ceiling is within 0.6 points of mSAP and RS@k on both datasets, while its higher efficiency accounts for its entire gain over mSAP.The analysis also finds that gSAP still leaves some achievable consistency unrealized, which per-query losses cannot optimize because they are blind to row interleaving.Source: E THEORETICAL ANALYSIS · p. 34
  • For a fixed set of per-query rankings, C is the maximum gAP attainable across order-preserving interleavings of the query lists.The efficiency eff = gAP/C measures the share of that ceiling realized by the model.Source: E THEORETICAL ANALYSIS · pp. 34–36
  • An optimal interleaving places each positive immediately after negatives from its own query, forming ordered blocks whose density is one positive per block item.Trailing negatives can be placed after all positives without affecting gAP.Source: E THEORETICAL ANALYSIS · p. 36
  • The block schedule merges adjacent blocks until density is non-increasing within each query chain, then orders segments by decreasing density across chains.The method uses Sidney's ratio rule, pool-adjacent-violators merging, and the probability ranking principle.Source: E THEORETICAL ANALYSIS · p. 36
  • On full-size runs, reported C—the larger gAP of the rule and greedy schedules—lies between that value and an upper bound, with gaps at most 7×10^-7 for SOP and 6×10^-6 for iNaturalist.Source: E THEORETICAL ANALYSIS · p. 36
  • Across 400 small synthetic instances, the ratio rule found the optimum in 382, and the reported value matched it in 393.On truncated real-query instances, the rule found the optimum in 99 of 100 SOP cases and 82 of 100 iNaturalist cases; reported values matched in 100 and 86 cases, respectively.Source: E THEORETICAL ANALYSIS · p. 36
Loading 2610.09863v1…