PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic Embeddings for Cross-Modal Retrieval

Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio de Rezende, Yannis Kalantidis, Diane Larlus

arXiv:2101.05068v2cs.CV

TL;DR

Cross-modal retrieval must handle image-caption pairs with multiple valid correspondences, which deterministic functions do not adequately capture. The paper proposes PCME, representing multimodal samples as probability distributions and evaluating retrieval with a cleaner CUB benchmark alongside COCO. PCME improves retrieval over its deterministic counterpart and supplies uncertainty estimates that make embeddings more interpretable.

  • Problem

    Image-caption retrieval contains one-to-many correspondences, while deterministic functions are not sufficiently powerful to capture them; COCO also has non-exhaustive cross-modal annotations.

  • Method

    PCME represents image and text samples as probability distributions in a shared embedding space and uses probabilistic contrastive learning for retrieval.

  • Results

    PCME improves retrieval performance over its deterministic counterpart and provides uncertainty estimates that render embeddings more interpretable.

  • Takeaways & Limitations

    Probabilistic embeddings model one-to-many image-caption associations while adding uncertainty information and richer structure to the joint embedding space.

Abstract

from arXiv · show

Cross-modal retrieval methods build a common representation space for samples from multiple modalities, typically from the vision and the language domains. For images and their captions, the multiplicity of the correspondences makes the task particularly challenging. Given an image (respectively a caption), there are multiple captions (respectively images) that equally make sense. In this paper, we argue that deterministic functions are not sufficiently powerful to capture such one-to-many correspondences. Instead, we propose to use Probabilistic Cross-Modal Embedding (PCME), where samples from the different modalities are represented as probabilistic distributions in the common embedding space. Since common benchmarks such as COCO suffer from non-exhaustive annotations for cross-modal matches, we propose to additionally evaluate retrieval on the CUB dataset, a smaller yet clean database where all possible image-caption pairs are annotated. We extensively ablate PCME and demonstrate that it not only improves the retrieval performance over its deterministic counterpart but also provides uncertainty estimates that render the embeddings more interpretable. Code is available at https://github.com/naver-ai/pcme

1. Introduction

Cross-modal retrieval commonly maps images and text into a shared space, but one-to-many correspondences challenge deterministic representations. PCME addresses this with probabilistic embeddings, while CUB provides a cleaner evaluation setting than non-exhaustively annotated COCO.

  • Cross-modal retrieval maps items from different modalities into a common embedding space for nearest-neighbour search.
  • Images can match multiple captions, and captions can correspond to multiple visual manifestations because the modalities encode content differently.
  • Deterministic one-to-one functions cannot adequately quantify these many-to-many relationships, motivating representations that capture multiplicity.
  • PCME represents samples as probability distributions, modeling uncertainty and implicitly supporting many-to-many matching without explicit detection-based representations.
  • PCME also provides uncertainty estimates, supports richer set-based operations, and complements deterministic retrieval systems.
  • COCO has non-exhaustive cross-modal annotations, so the paper proposes the smaller CUB benchmark with cleaner matching annotations and evaluation metrics.

2. Related work

Prior work represents multiplicity with multiple global, region-level, or reasoning-based embeddings, often increasing computational cost. PCME instead uses probabilistic embeddings for multimodal retrieval, extending probabilistic representation beyond prior applications.

  • Earlier cross-modal retrieval methods learned joint image-sentence metric spaces using techniques including CCA, hinge-ranking losses, and unimodal structure preservation.
  • Single global representations struggle with diverse semantic concepts, motivating multiple image or sentence representations and region-word matching.
  • Cross-modal attention and reasoning modules achieve strong retrieval results but require query-database pairs to pass through additional test-time computation.
  • Probabilistic representations have been used in areas such as word embeddings and zero-shot recognition, with related methods applying distribution alignment for classification.
  • PCME differs by using a probabilistic contrastive loss whose latent features can be used directly for multimodal retrieval.

3. Method

PCME learns joint visual-textual embeddings that represent each image and caption as distributions, enabling retrieval to model multiple cross-modal matches and associated uncertainty.

  • Task definition: PCME defines cross-modal retrieval over image-caption datasets where a query may have multiple ground-truth matches.This multiplicity is the central focus of the method.
  • Joint visual-textual embeddings: Visual and textual encoders map images and captions into a shared R^D embedding space using modality-specific feature extractors and head modules.The visual branch uses ResNet features, while the textual branch uses GloVe features followed by a bidirectional GRU.
  • Probabilistic learning: PCME trains these distributions with a soft probabilistic contrastive loss based on Monte Carlo estimates of cross-modal match probabilities.Sample-wise match probabilities are computed from Euclidean distances using learnable scalars and a sigmoid function.
  • Probabilistic embeddings: Instead of point embeddings, PCME represents each sample as a normal distribution with mean vectors and diagonal covariance matrices in R^D.Separate head modules compute the mean and variance vectors for each modality.
  • Uncertainty modeling: The mean and variance heads differ because sigmoid, LayerNorm, and L2 projection are omitted from the uncertainty module to avoid overly restricted uncertainty estimates.Additional KL-divergence and uniformity losses regularize the learned distributions, while covariance volume provides a scalar uncertainty measure.
  • Multiplicity: For sampled embedding pairs, softmax weights emphasize distance-consistent matches, encouraging diverse samples and non-zero predicted variances.Positive pairs weight closer samples more heavily, whereas negative pairs weight more distant samples more heavily.

4. Experiments

The experiments evaluate PCME using improved cross-modal benchmarks and metrics, compare probabilistic distance variants with prior methods, and analyze uncertainty in the learned embedding space.

  • Benchmark evaluation: R-Precision is perfect if and only if all positive items are retrieved before negatives, while Plausible Match metrics expand positives using class-based similarity.The expanded criteria use ζ ∈ {0, 1, 2} and average the results.
  • Benchmark evaluation: COCO assigns exactly five matching captions per image and one matching image per caption, labeling other plausible pairs as negatives.This creates noisy training and unreliable evaluation results.
  • CUB results: 26.3% i2t and 26.8% t2i are achieved by match-probability similarity, outperforming deterministic PCME and trading higher performance for additional computation.The 2-Wasserstein variant reaches 26.2% and 26.7% while requiring twice the original space consumption.
  • CUB results: +1.6 pp and +1.2 pp are PCME’s gains over PCME µ-only in CUB i2t and t2i R-Precision, respectively.PCME provides the best performances on both R-Precision and R@1 except CUB i2t R@1.
  • COCO results: 34.1% versus 31.8%, 29.7%, and 33.0% is PCME’s 5K i2t PMRP result against prior COCO methods.The comparison indicates better retrieval of plausible matches under PMRP, although several prior methods perform better on R@1.
  • Embedding analysis: Increasing uncertainty correlates with lower R@1 performance on the COCO test set.PCME measures instance-wise uncertainty using the geometric mean of the σ entries and evaluates performance across ten uncertainty bins.
  • Embedding analysis: The generic CUB caption’s red embedding region covers most caption variations, illustrating how probabilistic embeddings represent broad caption uncertainty.The visualization uses ellipses as confidence regions for image and caption embeddings.

5. Conclusion

The paper introduces PCME as a probabilistic representation for multimodal data and evaluates its ability to model one-to-many image-caption associations. It reports improved retrieval over deterministic embeddings alongside uncertainty estimates that make embeddings more interpretable.

  • PCME learns probabilistic multimodal representations to model widespread one-to-many associations in image-caption pairs.
  • PCME improves retrieval performance over its deterministic counterpart while providing uncertainty estimates that render embeddings more interpretable.

Supplementary Materials

The supplementary materials detail PCME, probabilistic distance choices, evaluation alternatives, and additional benchmark protocol decisions.

  • The supplementary document covers PCME details, probabilistic distances, experimental protocols, ablations, and additional results.
  • PCME details: PCME supervision connects to PVSE by replacing one-hot best-match weights with a softmax scheme over candidate representations.
  • Probabilistic distances: Probabilistic distances include probability product kernels, Wasserstein distance, KL divergence, and Jensen-Shannon divergence for Gaussian embeddings.
  • Probabilistic distances: KL divergence is asymmetric and can hardly retrieve gallery embeddings with nearly zero variance; JS divergence can become numerically unstable at very small variances.
  • Evaluation: For CUB, pairs are positive when image and caption share a bird class; for COCO, positivity uses class-label Hamming distance and varying ζ criteria.
  • Evaluation: Because about half of COCO validation images have more than two categories, the PMRP evaluation uses ζ = 2 to reduce penalties from negligible objects.

C.2. Implementation details

The implementation uses pretrained visual and textual encoders, dataset-specific backbones and dimensions, staged training, and selected CUB bird classes.

  • Visual and textual encoders are initialized with ImageNet-pretrained ResNet and pretrained GloVe, then warm-started and fine-tuned end to end.
  • MS-COCO uses ResNet-152 with D = 1024, whereas CUB uses ResNet-50 with D = 512.
  • MS-COCO training contains 113,287 images and validation contains 5,000 images, with evaluation on 5K and averaged five-fold 1K test sets.
  • Hyperparameters are selected using 150 CUB training classes and the MS-COCO caption validation split.
  • The CUB caption experiments select nine bird classes spanning swimming birds, small birds, and woodpeckers.
  • A two-dimensional PCME variant removes attention modules from the image encoder and uses two caption projections.

D. Ablation studies

The ablations examine uncertainty regularization, variance flexibility, architecture, training samples, and embedding dimensions, generally favoring richer probabilistic configurations.

  • The ablations vary regularization terms, σ architectures, training sample count J, and embedding dimension D.
  • Regularizing uncertainty: The KL loss raises average sigma values from e^-13.01 ± 2.2×10^-6 to e^-3.84 ± 0.02, while uniformity prevents uncertainty collapse and slightly improves performance.
  • Variance flexibility: Allowing greater degrees of freedom for σ improves PCME R-Precision, compared with cheaper variance parameterizations.
  • Architecture study: Applying local attention to both µ and σ performs best, while sigmoid and layer normalization restrictions harm final performance.
  • Number of samples during training: Larger training sample counts J produce higher cross-validated mean R-Precision scores; J = 7 is selected under computation constraints.
  • Embedding dimensions: Across embedding dimensions, PCME consistently outperforms deterministic PCME µ only.

E. More results

This section provides additional PCME results on CUB Caption and COCO Caption, including comparisons of distributional pairwise distances and retrieval strategies.

  • Additional experiments evaluate PCME on CUB Caption and COCO Caption.
  • Pairwise distributional distances are compared by space complexity and retrieval performance using R-Precision.
  • Different retrieval strategies are compared in Figure E.1.

E.1. More results on similarity measures for retrieval at test time

The appendix compares similarity and negative-mining strategies for retrieval, showing that hardest negative mining can improve Recall@1 while reducing global plausibility measured by R-Precision.

  • Hardest negative mining considers only the most similar positive and negative pairs, neglecting other possible positive candidates.
  • The HNM comparison is motivated by measuring both R-Precision and Recall@1 to expose disadvantages for learning global structure.
  • PVSE with HNM raises Recall@1 from 36.3 to 37.6 to 41.1 as K increases, but lowers R-Precision from 21.4 to 20.4 to 19.2.
  • Non-HNM PVSE achieves lower Recall@1 than HNM for every K but higher R-Precision.

E.3. Full results for CUB and COCO

Full CUB and COCO results evaluate retrieval with R-Precision, PMRP, and Recall@K, while additional analyses examine how uncertainty and plausible matching behave across methods and perturbations.

  • CUB Caption: PCME achieves the best R-Precision against baselines on both CUB Caption unseen 50-class and seen 150-class splits.
  • COCO Caption: On MS-COCO, PCME slightly underperforms PVSE K=2 on R@1 but surpasses PVSE K=2 on PMRP.
  • COCO Caption: As ζ increases above zero, PCME’s PMRP improves, whereas VSRN and VSRN + AOQ show worsening PMRP.
  • Uncertainty analysis: Increasing image occlusion raises uncertainty levels, and uncertainty is also evaluated while appending unknown tokens to captions.
  • Uncertainty analysis: TF-IDF frequencies are shown by ascending uncertainty bin for image and caption modalities, using ground-truth captions for image-word frequencies.

F.3. Example uncertain samples

The appendix visualizes uncertain images and captions alongside PCME’s retrieved items and their ground-truth matches. These examples are used to emphasize the limits of Recall@1 and the relevance of plausible matching.

  • Uncertain images and captions are displayed with their corresponding PCME-retrieved items.
  • The retrieved captions and images are described as plausible results for their query items, even when Recall@1 may be noisy.
  • Figure F.2 organizes TF-IDF-sorted frequent words into ascending uncertainty bins for images and captions.
  • Example query content includes skateboarding, tennis, and surfing scenes paired with alternative descriptions or visual matches.
  • The uncertain-image examples include retrieved captions and ground-truth captions, while uncertain-caption examples include retrieved images and ground-truth images.
Loading 2101.05068v2…