PaperlayerView exact revision on arXiv

Source-linked AI summary

Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval

Yale Song, Mohammad Soleymani

arXiv:1906.04402v2cs.CV

TL;DR

Cross-modal retrieval is challenged by polysemous instances and partial associations, which single-point injective embeddings cannot represent well. The paper introduces PIE-Nets and combines them with multiple-instance learning for image-text and video-text retrieval, reporting strong results across three datasets and a state-of-the-art MS-COCO image-to-text result.

  • Problem

    Polysemous instances and partial cross-modal associations challenge retrieval methods that represent each instance with a single embedding.

  • Method

    PIE-Nets compute multiple representations by combining global and locally guided features, while two tied PIE-Nets are jointly trained with multiple-instance learning.

  • Results

    The approach reports strong performance on MS-COCO, TGIF, and MRW, including a state-of-the-art image-to-text retrieval result on MS-COCO.

  • Takeaways & Limitations

    The work extends polysemous visual-semantic embedding to image-text and video-text retrieval and introduces MRW for video-text retrieval under ambiguity and partial association.

  • Takeaways & Limitations

    A more sophisticated multi-head self-attention variant did not improve performance further, and diversity loss is not applied to final representations because their global information makes orthogonality invalid.

Abstract

from arXiv · show

Visual-semantic embedding aims to find a shared latent space where related visual and textual instances are close to each other. Most current methods learn injective embedding functions that map an instance to a single point in the shared space. Unfortunately, injective embedding cannot effectively handle polysemous instances with multiple possible meanings; at best, it would find an average representation of different meanings. This hinders its use in real-world scenarios where individual instances and their cross-modal associations are often ambiguous. In this work, we introduce Polysemous Instance Embedding Networks (PIE-Nets) that compute multiple and diverse representations of an instance by combining global context with locally-guided features via multi-head self-attention and residual learning. To learn visual-semantic embedding, we tie-up two PIE-Nets and optimize them jointly in the multiple instance learning framework. Most existing work on cross-modal retrieval focuses on image-text data. Here, we also tackle a more challenging case of video-text retrieval. To facilitate further research in video-text retrieval, we release a new dataset of 50K video-sentence pairs collected from social media, dubbed MRW (my reaction when). We demonstrate our approach on both image-text and video-text retrieval scenarios using MS-COCO, TGIF, and our new MRW dataset.

1. Introduction

Visual-semantic embedding maps visual and textual instances into a shared space, but injective single-point representations struggle with ambiguous meanings and partial cross-modal associations. The paper addresses these issues with multiple instance representations and a partial-match training objective, evaluating image-text and video-text retrieval.

  • Visual-semantic embedding learns mappings from visual and textual domains into a shared embedding space for cross-modal applications.The paper identifies tagging, retrieval, captioning, and visual question answering as downstream applications.
  • Injective embedding forces an instance with multiple meanings or senses into a single point, potentially producing an inaccurate weighted geometric mean.The problem becomes more severe for videos and sentences because ambiguity can accumulate across images, words, or frames.
  • Partial associations arise when text describes only some image regions or video frames, leaving the relevant cross-modal parts implicit.Ignored content can therefore be incorporated into the mapped representation even when it is not described by the paired instance.
  • PIE-Net addresses ambiguity by extracting K diverse embeddings that combine global information with locally guided representations obtained through multi-head self-attention and residual learning.The local representations can attend to regions, frames, or words, while residual fusion discourages redundant information.
  • PVSE ties two PIE-Nets and uses multiple-instance learning so that only one of K×K embedding pairs must align, while MMD regularizes distributional discrepancy.This relaxes the perfect-association constraint and addresses the unconstrained remaining embedding pairs.
  • The approach is evaluated on image-text retrieval with MS-COCO and video-text retrieval with TGIF and the newly collected MRW dataset.MRW contains 50K video-sentence pairs from social media, and the authors report strong performance on all three datasets, including a state-of-the-art MS-COCO image-to-text result.

2. Related Work

Prior cross-modal retrieval methods largely use one-to-one embeddings, correlation or triplet objectives, auxiliary tasks, or context-dependent attention. The paper instead extends multi-representation embedding to both modalities and combines global and locally guided features through residual learning.

  • Correlation maximization: Correlation-maximization methods learn shared embeddings by maximizing relationships between related instances, including through CCA and its triplet extensions.
  • Triplet ranking: Triplet-ranking methods bring positive pairs closer than negative pairs, with hard-negative mining focusing on the maximum-violating negatives.
  • Learning with auxiliary tasks: Other approaches jointly learn embeddings with auxiliary tasks such as signal reconstruction, semantic categorization, distribution matching, or adversarial domain alignment.
  • Attention-based embedding: Attention-based methods selectively attend to local parts of one instance conditioned on another modality, but remain based on one-to-one mapping.This conditioning can address ambiguity and partial association by selecting different local content for different queries.
  • Beyond injective embedding: Stochastic embedding maps instances to probability distributions rather than points, but distribution learning is typically difficult or expensive and may require approximate sampling.
  • Beyond injective embedding: Unlike prior multi-representation work, this method computes multiple representations for both modalities, combining global context with locally guided features via residual learning.It also extends MRW and adds MS-COCO image-text cross-retrieval experiments.

3. Approach

PVSE uses two independent PIE-Nets to produce multiple embeddings from global context and locally guided features, then trains them with objectives designed for partial associations and diverse representations. Its optimization combines multiple-instance learning with diversity and domain-discrepancy regularization.

  • Architecture: PVSE comprises modality-specific feature extractors followed by two independent PIE-Nets with identical architectures.The two PIE-Nets do not share weights.
  • PIE-Net: Each PIE-Net transforms global context and local features into K embeddings through attention-guided representations and feature fusion.The local feature transformer uses multi-head self-attention to attend to different input parts, while fusion combines the resulting features with global context.
  • Feature Fusion With Residual Learning: Residual learning treats global features as input and locally guided features as residuals, encouraging the latter to capture locally specific information.If the residuals become zero, the model falls back to standard injective embedding.
  • MIL Loss: The MIL objective requires only one of K×K cross-modal embedding pairs to align, allowing mismatching pairs from partially associated instances to be ignored.The minimum-distance formulation also acts as hard negative mining and was found to accelerate convergence.
  • Diversity Loss: The diversity loss penalizes redundancy among K locally guided features using Gram matrices whose off-diagonal entries encode feature correlations.It is applied to locally guided features rather than final embeddings because the latter already contain global information.
  • Domain Discrepancy Loss: A domain discrepancy loss regularizes the distributions of visual and textual embeddings because MIL constrains only the minimum-distance pair.The paper uses Maximum Mean Discrepancy to measure the discrepancy between the two distributions and an RBF kernel for the computation.

4. MRW Dataset

MRW is a 50K-scale social-media video-sentence dataset built around reaction videos responding to situations described in text. Its weaker, inherently ambiguous video-text associations create a challenging retrieval setting.

  • Dataset overview: MRW contains 50K video-sentence pairs collected from social media to support research on ambiguous and partially associated video-text retrieval.The dataset is called MRW (my reaction when).
  • Dataset characteristics: Unlike captioning datasets, MRW videos show physical or emotional reactions to situations described by sentences rather than depicting the sentences’ visual content directly.This creates a text →video relationship instead of the usual video →text relationship.
  • Dataset characteristics: A reaction GIF is an animated GIF containing a physical or emotional response to something described externally, not to an event occurring within the GIF.The latter case is classified as a scene rather than a reaction GIF.
  • Dataset characteristics: Figure 4 presents four common reaction types: physical, emotional, animal, and lexical.The figure illustrates the kinds of reactions represented in the dataset.

5. Experiments

The experiments evaluate PVSE on image-text and video-text retrieval using MS-COCO, TGIF, and MRW, with standard retrieval metrics and ablations of its embedding and training design. Results show strong benchmark performance, benefits from multiple embeddings and balanced global-local features, and greater difficulty on MRW.

  • Experimental setup: PVSE is evaluated on MS-COCO for image-text retrieval and TGIF and MRW for video-text retrieval using Recall@k, Med R, nMR, and rsum.Model selection uses the highest validation rsum across both retrieval directions.
  • Image-text retrieval results: PVSE achieves the new state-of-the-art on MS-COCO image-to-text retrieval on the 5K test set and outperforms most baselines.It uses a single ranking objective, unlike two compared methods trained with multiple objectives.
  • Video-text retrieval results: PVSE shows strong performance against all baselines on both TGIF and MRW, where no previously reported cross-modal retrieval results were available.The authors therefore run the baseline models themselves for comparison.
  • Video-text retrieval results: Video-text retrieval performance is much lower than MS-COCO performance, reflecting the greater challenge of video-text retrieval and differing dataset characteristics.TGIF sentences describe video content, whereas MRW videos show possible reactions to situations described in sentences.
  • Ablation results: K = 3 is optimal on COCO and TGIF, while K = 5 is optimal on MRW; performance improves significantly from K = 0 to K = 1.K = 0 bypasses the PIE-Net and uses the global feature directly; the authors cautiously associate MRW’s larger optimum with greater ambiguity.
  • Ablation results: Discarding global features or locally-guided features reduces performance, while replacing residual learning with concatenation hurts especially on MRW.The results support balancing global and local information and using residual learning for ambiguous data.
  • Ablation results: The MIL objective significantly outperforms the No MIL setting on MS-COCO and MRW, with p = 0.046 and p = 0.015, respectively.The difference is more pronounced on MRW than on MS-COCO.
  • Ablation results: Both loss terms are important, Lmmd is particularly important, and performance is not highly sensitive to their relative weights.Without Lmmd, the modality-specific embedding spaces may diverge under the MIL objective.

6. Conclusion

The paper addresses ambiguity and partial cross-modal association by replacing single-point embeddings with multiple representations and training paired PIE-Nets using MIL. It demonstrates strong results across image-text and video-text retrieval and contributes the ambiguous MRW benchmark.

  • Conclusion: PIE-Net computes multiple and diverse representations per instance, while paired PIE-Nets trained with MIL improve robustness to ambiguous and partial associations.The approach targets cross-modal retrieval rather than only one modality pair.
  • Conclusion: MRW contributes a dataset where videos illustrate possible reactions to situations described in sentences, creating weaker and more ambiguous associations than conventional video-caption datasets.The dataset is intended to pose new challenges for cross-modal retrieval.

A. MRW Dataset

MRW is a continuously expanding collection of social-media reaction videos paired with user-written sentences. Its animated-GIF format and real-world sourcing distinguish it from conventional video-caption datasets.

  • Collection: MRW contains 50,107 video-sentence pairs crawled from Reddit, Imgur, and Tumblr through the GIPHY API between August 2016 and March 2019.The authors plan to release updated versions as crawling continues.
  • Format: The dataset primarily uses animated GIFs, which the paper treats interchangeably with videos because their format distinction is unnecessary for the method.Animated GIFs are lossless, palette-based, and lack audio.
  • Related work: Animated GIFs have been studied for engagement and used in computer vision tasks including video understanding, captioning, and visual question answering.Prior work identifies animation, immediacy, low bandwidth, and emotional expression as relevant properties.
  • Distinctive design: Unlike study-participant caption datasets, MRW pairs videos with sentences written by real Internet users, introducing real-world challenges and implicit concept associations.Its videos represent reactions to situations described in text rather than direct visual descriptions.

A.2. Analysis of Facial Expressions

The dataset analysis measures facial presence and expression intensity in animated GIFs. Faces appear frequently, with joy and disgust the most common detected expressions.

  • 72% of the videos contain at least one face, averaging 0.73 faces per frame.Face detection sampled five equally spaced frames from each animated GIF.
  • Joy has the highest average expression intensity at 9.1%, followed by disgust at 7.4%.Expressions were analyzed on face-containing frames using six basic emotions.
  • The analysis covers joy, fear, sadness, disgust, surprise, and anger using frames whose face bounding box exceeds 15% of the image.Expression intensity was detected at two frames per second in each animated GIF.

A.3. Comparison to the TGIF Dataset

MRW differs from TGIF because its sentences and video-text associations are more implicit and ambiguous. Human judgments and language distributions both indicate that MRW poses a harder retrieval setting.

  • MRW contains physical or emotional reactions to situations in sentences, whereas TGIF sentences explicitly describe visible video content.This distinction creates a weaker video-text concept association in MRW.
  • MRW uses more abstract nouns and verbs, including “time,” “realize,” and “think,” than TGIF.These terms are less directly visualizable and contribute to ambiguous associations.
  • 80.4% of MRW associated pairs were marked relevant, while 50.7% of randomly assigned pairs were also marked matching.The high false-positive rate indicates ambiguous GIF-sentence associations despite humans distinguishing true from fake pairs.
  • TGIF achieved 95.2% relevance for positive pairs and 2.6% relevance for irrelevant pairs in the human study.The contrast provides a human baseline for MRW’s more challenging implicit associations.
  • MRW was collected to support text-based animated GIF search under nuanced, implicit visual-textual associations.The dataset contains 50K video-sentence pairs from social media.

B. Baseline Implementation Details

The baselines use shared video and sentence encoders but differ in their training losses. Additional analyses compare loss modifications and attention-based representations for cross-modal retrieval.

  • All four baselines share the same video and sentence encoders, differing only in their loss functions.The implementations cover DeViSE, VSE++, Order Embedding, and Corr-AE.
  • DeViSE uses conventional hinge-loss triplet ranking with cosine distance and margin ρ.The loss penalizes positive pairs that are farther apart than negative pairs.
  • VSE++ uses hard-negative triplet ranking with a z-score filter that retains multiple hard negatives.The modification addresses the strict max constraint under noisy TGIF and MRW data.
  • Corr-AE trains modality-specific autoencoders to reconstruct both video and sentence encoder outputs.The reported implementation uses four fully connected layers with ReLU activations and layer normalization.
  • Using L2 instead of the DeViSE loss made Corr-AE substantially harder to train and produced results nearly identical to random guessing.This differs from the original Corr-AE formulation described by the authors.
  • PIE-Net attention maps focus on different image regions, combining global context with region-specific locally-guided features.Examples show global information capturing a main object while local features represent surrounding regions.

C.2. Video-to-Text Retrieval Results on TGIF

On TGIF video-to-text retrieval, visual and textual attention maps generally highlight salient frames and words. Retrieved sentences tend to reflect the concepts emphasized by their corresponding visual attention.

  • Visual and textual attention maps tend to highlight salient video frames and words, respectively.The qualitative examples show alignment between attended temporal regions and retrieved sentence content.
  • Retrieved results commonly capture concepts highlighted by their corresponding visual attention maps.For example, one result focuses on scoring a goal, while another also captures a player being hit by the ball.
  • Different temporal attention patterns correspond to different retrieved descriptions of the same video.The examples distinguish retrievals by the event or action emphasized in the attended frames.

C.3. Text-to-Video Retrieval Results on MRW

MRW retrieval results illustrate that visual-textual associations are ambiguous, with multiple retrieved videos often plausibly matching a query. This ambiguity motivates evaluating perceptual similarity rather than relying only on exact-match retrieval metrics.

  • Figure 12 presents text-to-video retrieval queries with the top five retrieved videos, their ranks, and cosine similarity scores, without directly displaying ground-truth matches.The presentation asks readers to identify the matches and emphasizes the dataset’s ambiguous and implicit visual-textual associations.
  • Most top-five retrieved videos appear to match their query, including videos sharing facial expressions, animals and food, or awkward dancing movements.The examples show that multiple videos can plausibly correspond to the same sentence.
  • Highly ambiguous associations and multiple correct matches partly explain the relatively poor reported retrieval performance.The authors argue that exact-match evaluation may not reflect the perceptual similarity between queries and retrieved results.
Loading 1906.04402v2…