Probabilistic Embeddings for Cross-Modal Retrieval

Introduction

• Image-caption retrieval requires modeling matches in both directions because images can fit multiple captions and captions can describe multiple visual manifestations.The different modalities contribute to this multiplicity: photographs capture scene components, while language selectively reports relevant concepts.Source: §1 · p. 1
• Vanilla embedding functions model one-to-one relationships [10] [54]; alternatives include PVSE’s K candidates and computationally costly region-word matching [27].PVSE was shown to capture multiplicity and improve over one-to-one baselines.Source: §1 · pp. 1–2
• PCME represents images and captions as probability distributions, modeling one-to-many matches without explicit many-to-many representations and providing uncertainty estimates.The paper proposes analyzing the joint embedding space through those estimates, which reveal intuitive properties.Source: Figure 1 · p. 1, §1 · p. 2
• PCME’s probabilistic representation supports uncertainty-based estimates of query difficulty or failure chance, set algebra, and complementarity with deterministic retrieval systems.The paper presents probabilistic mapping as an effective alternative to explicit many-to-many representations such as detection-based approaches.Source: §1 · p. 2
• Because MS-COCO annotations are non-exhaustive and exhaustive manual annotation does not scale, the paper proposes CUB as a smaller, cleaner benchmark with more sensible metrics.The paper identifies shortcomings in existing cross-modal retrieval benchmarks and proposes alternative solutions.Source: §1 · p. 2

Cross-modal retrieval

• Joint image-sentence metric spaces have been learned with CCA [25] and ranking losses, including hard-negative triplet training [10].Other approaches use a hinge rank loss or add unimodal triplets to preserve modality-specific structure [54].Source: §2 · p. 2
• To address one-to-many matching, prior work uses multiple global representations, region- and word-level matching, or conditional embeddings, but the latter lack a joint retrieval space.Examples include multi-head self-attention, graphical convolution over detected regions [27], and conditional embeddings for hashtag multiplicity [52].Source: §2 · p. 2
• Cross-modal attention reasoning achieves state-of-the-art retrieval but requires query-database pairs to pass through a test-time module, so this work favors scalable joint embeddings.The pairwise reasoning cost grows with evaluating the query against every database entry, whereas joint embedding approaches support large-scale indexing.Source: §2 · p. 2

Probabilistic embedding

• Probabilistic embeddings span word representations [53] and vision tasks; HIB [39] applies them to one-to-many metric learning, which PCME extends across image and caption domains.Prior vision applications also include face understanding [4] [46], 2D-to-3D pose estimation [49], speaker diarization [47], and prototype embeddings [45].Source: §2 · p. 2
• PCME extends HIB [39] to joint image-caption embeddings, uses probabilistic contrastive loss for direct retrieval, and is presented as the first probabilistic embedding method for multimodal retrieval.Unlike the conceptually similar latent space of Sch6nnfeld et al., PCME uses its probabilistic features directly for retrieval rather than aligning distributions with 2-Wasserstein distance and training classifiers.Source: §2 · pp. 2–3

3.1. Building blocks for PCME

• PCME combines joint visual-textual embeddings with probabilistic embeddings to address multiple valid cross-modal matches.For a given image or caption, the corresponding set may contain multiple matches.Source: §3 · p. 3, §3.1 · p. 3

3.1.1 Joint visual-textual embeddings

• Visual and textual encoders produce shared-space representations, while PVSE models one-to-many matches with multiple attention-generated embeddings and best-pair supervision.The encoders use ResNet [15] for images and GloVe features with a bidirectional GRU [7] for captions; PVSE supervises only the best embedding pair.Source: §3.1.1 · p. 3

3.1.2 Probabilistic embeddings for a single modality

• PCME builds on HIB [39], which maps inputs to distributions and uses a soft contrastive loss to learn match probabilities while representing data uncertainty.The loss penalizes low match probability for positive pairs and high match probability for negative pairs.Source: §3.1.2 · p. 3

3.2.1 Model architecture

• PCME represents images and captions as normal distributions with mean and diagonal covariance vectors, using separate heads to predict the distribution parameters.Its local-attention branch aggregates spatial features; the variance head omits sigmoid, LayerNorm, and L2 projection, which overly restrict uncertainty estimates. KL-divergence and uniformity losses regularize uncertainty, whose scalar measure is covariance volume.Source: Figure 2 · p. 4, §3.2.1 · pp. 4–5

3.2.2 How does our loss handle multiplicity, really?

• PCME’s loss weights positive pairs more when their distances are smaller and negative pairs more when their distances are larger, rewarding correct distance relationships.This reward allows some incorrect similarity predictions to go unpenalized when at least one prediction is correct, encouraging diverse samples and non-zero variance estimates.Source: §3.2.2 · p. 5

3.2.3 Test-time variants

• At test time, PCME can estimate match probabilities by sampling, using means alone, or using closed-form Gaussian distances such as 2-Wasserstein, Jensen-Shannon, and ELK.With J Monte Carlo samples, sampling requires O(J^2) distance computations per match and O(J^2) space per database entry; means-only scoring has O(1) time and space complexity.Source: §3.2.3 · p. 5

4.1. Experimental protocol

• Experiments use pretrained ResNet and GloVe encoders, warm up head modules before end-to-end fine-tuning, and train with Cutout and random caption dropping.The COCO and CUB backbones are ResNet-152 and ResNet-50, respectively; both use AdamP with a cosine learning-rate scheduler.Source: §4.1 · p. 5

4.1.1 Metrics for cross-modal retrieval

• R-Precision is perfect only when all positives rank ahead of negatives, so the paper defines PMRP using class-based plausible matches and averages over ζ values 0, 1, and 2.The passage notes that lenient Recall@k can fail to penalize obviously wrong top-k samples; R-Precision requires exhaustive positive annotations.Source: §4.1.1 · p. 6

4.1.2 Cross-modal retrieval benchmarks

• COCO’s fixed binary labels treat only five captions per image and one image per caption as matches, creating noisy training and unreliable evaluation; CUB Captions is proposed as a cleaner surrogate.CUB uses ten captions per image across 200 bird categories, with homogeneous within-class pairs and class-distinguishing annotations intended to reduce false positives and negatives.Source: §4.1.2 · p. 6

4.2. Results on CUB

• For PCME retrieval, match-probability similarity attains the best reported results, 26.3% i2t and 26.8% t2i, versus 2-Wasserstein’s 26.2% and 26.7%, respectively.The passage describes a trade-off between computational cost and performance. 2-Wasserstein introduces only twice the original space consumption.Source: §4.2 · p. 6
• On CUB, PCME achieves the best R-Precision and R@1 results except for i2t R@1, and improves over PCME µ-only by 1.6 pp and 1.2 pp in i2t and t2i R-Precision.PVSE with HNM can improve R@1 while scoring below VSE0 on R-Precision, suggesting its gains favor diversity over precision.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. Results on COCO

• 34.1% PMRP for PCME on the 5K COCO i2t task exceeds the cited prior methods’ 31.8%, 29.7%, and 33.0%, although several methods score higher on R@1.PCME’s stochastic version also outperforms its deterministic variant across the reported results.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Understanding the learned uncertainty

• Higher embedding uncertainty accompanies lower COCO R@1, while erasing image pixels or caption words increases σ, linking uncertainty to ambiguity.The study bins queries by geometric-mean σ and reports declining average R@1 across uncertainty bins.Source: §4.4 · pp. 7–8, Figure 6 · p. 8, Figure 7 · p. 8
• In the CUB embedding visualization, mixing two images maps roughly to the intersection of their embeddings, while erasing a small region widens distributions, suggesting inclusion.A generic caption embedding covers most caption variations in the visualization.Source: §4.4 · pp. 7–8, Figure 8 · p. 8, Figure 5 · p. 8

• Extensive ablations show that PCME improves retrieval over its deterministic counterpart and provides uncertainty estimates that render embeddings more interpretable.Source: §5 · p. 8
• PCME learns probabilistic representations in a shared embedding space to model widespread one-to-many associations in image-caption pairs.Source: §5 · p. 8
• The authors describe this as the first work to use probabilistic embeddings for a multimodal task.Source: §5 · p. 8

A. More details for PCME

• The supplementary materials add PCME details, probabilistic distances, experimental protocol and ablations, and further retrieval results and uncertainty analyses.They also provide additional materials to complement the main paper.Source: Supplementary Materials · p. 10, A. More details for PCME · p. 10

A.1. The uniformity loss

• Because PCME’s mean vectors lie on the unit L2 hypersphere, the method applies Wang et al.’s [55] uniformity loss to joint embeddings to encourage better representations.Source: §A.1 · p. 11

A.2. Connection between the soft contrastive loss and the MIL objective of PVSE

• With one-hot weights selecting only the closest positive and negative match, PCME’s soft contrastive loss recovers PVSE’s MIL loss, while the softmax scheme gives more interpretable and performant uncertainty supervision than argmax.This connection is presented alongside the paper’s gradient-based analysis of how the loss handles plurality in matches and learns data uncertainty.Source: §A.2 · p. 11

B. Probabilistic distances

• The supplementary distance variants compare normal distributions: KL is asymmetric and fails the triangle inequality, while the square root of JS is a metric; both KL and JS can be numerically unstable at very small variances.A near-zero-variance gallery embedding is difficult to retrieve with KL; the paper says KL performs poorly in real-world retrieval.Source: B. Probabilistic distances · p. 11
• The materials also describe probability product kernels, including ELK at ρ=1 and Bhattacharyya kernel at ρ=1/2, alongside the 2-Wasserstein distance for normal distributions.The Bhattacharyya kernel is also called Bhattacharyya’s affinity; probability product kernels are cited as.Source: B. Probabilistic distances · p. 11

C. Experimental Protocol Details

• The experimental protocol section introduces the retrieval benchmarks and discusses shortcomings of current evaluation practice and alternative evaluation approaches.Source: C. Experimental Protocol Details · p. 11, §C.1 · p. 12

C.1. Plausible Match R-Precision (PMRP) details

• For PMRP, CUB pairs are positive when they share a bird class, while COCO pairs are positive when their 80-class vectors differ in at most ζ positions; ζ=2 avoids penalizing negligible objects.Forty-eight MS-COCO 5k test images without instance labels are omitted. About half of validation images have more than two categories; results for other ζ values appear in §E.Source: §C.1 · p. 12, Figure C.1 · p. 12

C.2. Implementation details

• ResNet [15] and GloVe initialize the encoders, which are first warmed up with frozen feature extractors and then fine-tuned end to end.The study uses ResNet-152 with D = 1024 for MS-COCO and ResNet-50 with D = 512 for CUB; all experiments use J = 7 samples, AdamP [16], and a cosine learning-rate scheduler [31].Source: §C.2 · p. 12
• MS-COCO evaluation follows [19], adds the validation set to the training pool, and reports results on the 5K and average-over-five-fold 1K test sets.The training and validation splits contain 113,287 and 5,000 images, respectively.Source: §C.2 · p. 12
• Hyperparameters are validated on the 150 CUB training classes and the MS-COCO caption validation split, with dataset-specific learning rates, warm-up epochs, and fine-tuning epochs.MS-COCO uses learning rate 0.0002 and 30 warm-up plus 30 fine-tuning epochs; CUB Caption uses 0.0001 and 10 plus 50 epochs.Source: §C.2 · p. 12
• Both datasets use Cutout [8] and random caption dropping [3] with erasing ratios of 0.2 and 0.1, respectively.Regularizer weights (LKL, LUnif) are (0.00001, 0) for MS-COCO and (0.001, 10) for CUB Caption; initial a, b values are (-15, 15) and (-5, 5), respectively.Source: §C.2 · p. 12

C.3. CUB 2D toy experiment details

• The CUB 2D toy experiment selects nine bird classes spanning swimming birds, small birds, and woodpeckers.The selected classes are Western Grebe, Pied Billed Grebe, Pacific Loon; Vermilion Flycatcher, Black And White Warbler, American Redstart; and Red Headed Woodpecker, Red Bellied Woodpecker, Downy Woodpecker.Source: §C.3 · p. 12
• For the CUB toy experiment, PCME is modified to learn 2D embeddings, and the image encoder omits attention modules from its µ and σ modules.The caption encoder is a 1024-dimensional bi-GRU over GloVe vectors with two 2D projections producing the 1024-dimensional µ and σ embeddings; other training details match the CUB caption experiments.Source: §C.3 · p. 12

Regularizing uncertainty

• The KL loss moves predicted σ values into a meaningful range, while uniformity loss prevents uncertainty collapse and slightly improves performance.The regularization comparison reports cross-validated MAP@R on the 150-class training CUB caption dataset.Source: D. Ablation studies · pp. 12–13, Table D.1 · p. 13

DoF for σ

• Allowing σ more degrees of freedom improves PCME’s R-Precision, and dimension-wise variances indicate that the learned variances require high degrees of freedom.The comparison varies σ parameterization from a scalar isotropic Gaussian to full diagonal covariance.Source: D. Ablation studies · p. 13, Table D.2 · p. 13, Figure D.1 · p. 13

Architecture study

• Applying local attention to both µ and σ performs best, while restricting σ representations with sigmoid and layer normalization harms performance.Source: D. Ablation studies · pp. 13–14, Table D.3 · p. 13

Number of samples during training

• Larger training sample counts J yield higher cross-validated mean R-Precision; J = 7 is selected to fit computation budgets.Source: D. Ablation studies · p. 14, Figure D.2 · p. 14

Embedding dimensions

• Across all tested embedding dimensions, PCME consistently outperforms the deterministic PCME µ-only approach.The comparison is reported for both PCME µ only and PCME across different embedding-space dimensions.Source: D. Ablation studies · p. 14, Figure D.3 · p. 14

• Additional PCME experimental results cover CUB Caption and COCO Caption.Source: E. More results · p. 14
• Table E.1 compares distribution-distance options by space complexity and retrieval performance, with R-Precision used as a retrieval metric.Source: Table E.1 · p. 14
• Figure E.1 compares different retrieval strategies.Source: Figure E.1 · p. 14

E.1. More results on similarity measures for retrieval at test time

• KL-divergence performs worse than the non-probabilistic “Mean only” baseline.The full retrieval results for the distribution distances are reported in Table E.1.Source: §E.1 · p. 14
• Matching-probability sampling outperforms non-sampling strategies from J = 3, and larger J yields better performance.Source: §E.1 · p. 14
• The authors use J = 7 in Table E.1 because of the computation complexity.Source: §E.1 · p. 14

E.2. Discussion on hardest negative mining

• Recent cross-modal retrieval methods optimize Recall@1 directly using hardest negative mining (HNM) [10].Source: §E.2 · p. 14
• HNM considers only the most similar positive and negative pairs, neglecting other possible positive candidates.The discussion measures R-Precision and Recall@1 on CUB caption to examine how mining affects global-structure learning.Source: §E.2 · p. 15
• With HNM, PVSE Recall@1 rose 36.3→37.6→41.1 as K increased, while R-Precision fell 21.4→20.4→19.2.Figure E.2 compares the mining strategies on CUB caption.Source: §E.2 · p. 15, Figure E.2 · p. 15
• For all K, non-HNM PVSE has lower Recall@1 but higher R-Precision than HNM PVSE; the same phenomenon is reported on MS-COCO using PMRP scores.Source: §E.2 · p. 15

E.3. Full results for CUB and COCO

• As ζ increases, VSRN and VSRN + AOQ PMRP declines, while PCME improves for ζ > 0 and retrieves more plausible items.Source: Figure E.3 · p. 15, §E.3 · p. 15
• PCME achieves the best R-Precision on both CUB Caption splits, covering unseen 50-class and seen 150-class test sets.Source: §E.3 · p. 15, Table E.3 · p. 15
• On MS-COCO 1k and 5k tests, PCME has slightly worse R@1 than PVSE K=2 but outperforms it on PMRP.Source: §E.3 · p. 15, Table E.4 · p. 16, Table E.5 · p. 16

F. More uncertainty analysis

• The additional uncertainty analyses examine image and caption corruption alongside frequent words grouped by uncertainty bin.Source: F. More uncertainty analysis · p. 16, §F.1 · p. 16, §F.2 · p. 16

F.2. Frequent words for each uncertainty bin

• Frequent words per uncertainty bin are counted using TF-IDF; image-word frequencies use ground-truth captions, with N total captions and n_i captions containing word i.Source: §F.2 · p. 16

F.1. Corruption vs. uncertainty in MS-COCO

• In MS-COCO, greater pixel occlusion leads to higher uncertainty; uncertainty is also evaluated as unknown tokens are appended to captions.Source: Figure F.1 · p. 16, §F.1 · p. 16

F.3. Example uncertain samples

• Figures F.3 and F.4 visualize highly uncertain images and captions alongside PCME-retrieved items and their ground-truth captions or images.Source: §F.3 · p. 16, Figure F.3 · p. 18, Figure F.4 · p. 18
• Figure F.2 shows TF-IDF-sorted word frequencies by ascending uncertainty bin for image and caption modalities.Source: Figure F.2 · p. 18
• The qualitative examples show plausible retrieved captions and images, and identify PMRP as a more plausible and reliable comparison measure than noisy Recall@1.The examples pair highly uncertain images or captions with PCME retrievals and their ground-truth matches.Source: §F.3 · p. 17, Figure F.3 · p. 18, Figure F.4 · p. 18