Source-linked AI summary
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
Andrej Karpathy, Armand Joulin, Li Fei-Fei
TL;DR
Bidirectional image–sentence retrieval requires detailed cross-modal grounding of visual and linguistic content. The paper embeds image objects and sentence dependency relations in a shared space, combining global ranking with fragment alignment. Experiments report improved retrieval, while explicit fragment alignments provide interpretable predictions.
Problem
Image–sentence retrieval requires identifying entities, attributes, and relationships and grounding them appropriately across complex visual scenes.
Method
The model embeds image objects and sentence dependency-tree relations in a common multimodal space and combines Global Ranking and Fragment Alignment Objectives.
Results
The model significantly improves image–sentence retrieval performance across the reported tasks and produces interpretable predictions through explicit inter-modal fragment alignment.
Takeaways & Limitations
Reasoning over both global images and sentences and their finer-grained fragments supports bidirectional retrieval with explicit cross-modal alignment.
Takeaways & Limitations
Fragment alignment can fail when a concept is not visually identifiable, its region is missed by the detector, or its mention is omitted from another image’s description.
Abstract
from arXiv · showhide
We introduce a model for bidirectional retrieval of images and sentences through a multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. In addition to a ranking objective seen in previous work, this allows us to add a new fragment alignment objective that learns to directly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments significantly improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions since the inferred inter-modal fragment alignment is explicit.
1 Introduction
The paper addresses bidirectional image–sentence retrieval, where detailed entities, attributes, and relationships must be grounded across modalities. It proposes fragment-level multimodal reasoning alongside global image–sentence matching.
- Bidirectional retrieval supports image search from natural-language queries and image description through sentence ranking.The task ranks withheld sentences for an image query and images for a sentence query.
- Successful retrieval requires identifying entities, attributes, and relationships in sentences and grounding them in complex visual scenes.The paper illustrates this challenge with a sentence describing a dog, tennis ball, swimming, and murky water.
- The model embeds image objects and sentence dependency relations in a shared space, explicitly reasoning about their latent correspondences.A fragment-level loss complements the traditional sentence–image ranking loss.
- The authors report dramatic improvements over state-of-the-art methods on Pascal1K, Flickr8K, and Flickr30K image–sentence retrieval.The evaluation covers three image–sentence retrieval datasets.
2 Related Work
Related work spans bidirectional image–sentence mapping, multimodal representation learning, and neural representations for images and language. Earlier approaches include kernel alignment, compact semantic representations, probabilistic multimodal models, and shared neural embeddings.
- Image Annotation and Image Search: Prior bidirectional image–sentence methods include kernel canonical correlation analysis and common meaning spaces.The cited approaches align images and sentences or represent both with a single object–action–scene triplet.
- Multimodal Representation Learning: The paper differs from prior whole-image and whole-sentence embeddings by representing finer-grained image and language fragments.Its fragments are objects and typed dependency-tree relations.
- Image Annotation and Image Search: Kernel canonical correlation analysis is described as difficult to scale because it requires kernels quadratic in the numbers of images and sentences.
- Multimodal Representation Learning: Other multimodal representation-learning approaches model joint image–sentence distributions with Deep Boltzmann Machines, log-bilinear models, and topic models.An autoencoder using a shared bottleneck is also cited for audio–video representations.
- Neural Representations for Images and Natural Language: Neural representation work uses convolutional networks for visual representations and neural models for word or n-gram representations.The paper positions its model within neural approaches connected to image pixels and raw word representations.
3 Proposed Model
The proposed model performs bidirectional image-sentence retrieval by embedding image objects and sentence dependency relations as fragments in a shared multimodal space. It combines global ranking with fragment alignment and infers latent cross-modal correspondences.
- Overview of Learning and Inference: The model retrieves relevant images from sentences and sentences from images by learning high scores for compatible image-sentence pairs and low scores otherwise.Training correspondences are discarded before evaluation on withheld data.
- Fragment Embeddings: Images are decomposed into detected objects, while sentences are decomposed into typed dependency-tree relations that serve as visual and linguistic fragments.The model uses object detections from an RCNN and dependency relations rather than individual words or bigrams.
- Fragment Embeddings: The model embeds image and sentence fragments in a common space, computes image-sentence similarity from pairwise fragment scores, and infers their latent alignment.Inner products between fragment embeddings provide similarity scores, which are aggregated into an image-sentence score.
- Objective Function: The full objective combines a Global Ranking Objective, a Fragment Alignment Objective, and regularization to enforce compatible image-sentence rankings and fragment correspondences.The two learning objectives provide complementary information to the network.
- Fragment Alignment Objective: The Fragment Alignment Objective encourages at least one image region to match each sentence fragment while suppressing matches in unrelated images.The assumption can fail when a fragment is not visually identifiable, its box is undetected, or another image contains the concept without mentioning it.
- Fragment Alignment Objective: Multiple Instance Learning replaces dense fragment alignment with latent alignment by treating regions in the associated image as a positive bag and regions in other images as negatives.This addresses cases where a sentence relation refers to only one of several detected image regions.
- Global Ranking Objective: The global image-sentence score averages thresholded pairwise fragment scores and uses a smoothing term n = 10 to reduce the advantage of short sentences.Scores are truncated at zero before aggregation.
4 Experiments
Experiments evaluate bidirectional image-sentence retrieval across Pascal1K, Flickr8K, and Flickr30K, including quantitative comparisons, ablations, and qualitative analyses of fragment alignment. The full model outperforms previous methods, while the results support complementary global and fragment objectives and the value of object-level representations.
- Experimental Setup: The model is evaluated on Pascal1K, Flickr8K, and Flickr30K, with each image paired with five independently collected sentences.The datasets contain 1,000, 8,000, and 30,000 images, respectively.
- Experimental Setup: The evaluation ranks withheld images and sentences bidirectionally using dense image-sentence similarities.For Pascal1K, the split is 800 training, 100 validation, and 100 testing images; Flickr datasets use 1,000 validation and 1,000 testing images.
- Quantitative Results: The full method consistently and significantly outperforms previous methods on Flickr8K and Flickr30K.Pascal1K results show SDT-RNN competitive on Image Search, while the quantitative results are reported in Tables 1–3.
- Ablation Analysis: Combining the Global and Fragment Alignment Objectives improves performance over either objective alone, indicating that they contribute complementary information.The Global Objective performs slightly better individually, possibly because it directly minimizes the ranking cost.
- Representation Analysis: Using object detections rather than only a single full-frame CNN representation consistently improves performance.The experiments also compare dependency-tree relations with a simpler bag-of-words baseline, with dependency relations performing better on Flickr8K.
- Qualitative Analysis: Qualitative analyses expose explicit fragment alignments between retrieved sentences and image detections, including errors caused by associating “blue person” with a blue flag.Additional triplet retrieval experiments suggest that the model can generalize to fine-grained visual properties beyond ImageNet detection classes.
5 Conclusions
The paper concludes that fragment-level multimodal embeddings and latent alignment improve bidirectional image-sentence retrieval while producing interpretable predictions. It identifies counting, spatial reasoning, and moving beyond bags of fragments as future directions.
- Conclusions: Fragment-level reasoning supports a new Fragment Alignment Objective that complements Global Ranking Objective for bidirectional image-sentence retrieval.The model embeds image and sentence fragments in a multimodal space and reasons about their latent inter-modal alignment.
- Conclusions: The model significantly improves retrieval performance over previous work and produces interpretable predictions through explicit fragment alignment.The conclusion attributes the improvement to reasoning over finer image and sentence fragments alongside the global level.
- Future Work: Future work will extend the model to counting, spatial relations among objects, and representations beyond bags of fragments.