Source-linked AI summary
Learning Deep Structure-Preserving Image-Text Embeddings
Liwei Wang, Yin Li, Svetlana Lazebnik
TL;DR
The paper tackles semantic comparison between images and text by learning a shared embedding space for cross-view tasks. It uses a nonlinear two-branch network trained with bidirectional ranking and within-view structure-preserving constraints, and reports state-of-the-art retrieval results with improvements over CCA for phrase localization.
Problem
Image-text applications need a shared representation that measures semantic similarity between visual data and text for retrieval and related cross-view tasks.
Method
A two-branch nonlinear neural network is trained with bidirectional cross-view ranking constraints and within-view structure-preserving constraints.
Results
The method considerably exceeds the state of the art on Flickr30K and MSCOCO retrieval and shows convincing improvements over CCA on Flickr30K Entities phrase localization.
Takeaways & Limitations
The simple, flexible architecture applies to varied visual and textual features and supports both retrieval and phrase localization.
Takeaways & Limitations
CCA’s comparison is constrained by unstable minibatch covariance estimates for stochastic optimization, while retrieval datasets lack direct supervision for multiple images sharing a sentence.
Abstract
from arXiv · showhide
This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a large margin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and text-to-image retrieval. Our method achieves new state-of-the-art results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.
1. Introduction
The paper addresses joint image-text representation by learning a shared space that supports cross-view comparison and retrieval. It proposes a nonlinear two-branch embedding with ranking and within-view structure-preserving constraints, reporting improvements on retrieval and phrase localization.
- Motivation: Joint embeddings map images and text into a shared low-dimensional latent space for direct semantic comparison and cross-view retrieval.The motivation covers applications including image captioning and visual question answering.
- Limitations of prior work: CCA-based methods maximize cross-view correlation but are difficult to scale because minibatch covariance estimates are unstable for stochastic optimization.The paper identifies this as a limitation when handling large amounts of data.
- Proposed method: The proposed model uses two nonlinear branches over image and text representations, with fully connected layers and ReLU nonlinearities.The branches can build on pretrained networks, off-the-shelf feature extractors, or jointly trained representations.
- Training objective: Training combines bidirectional cross-view ranking with constraints that preserve neighborhood structure among semantically similar images or sentences.The within-view constraints are inspired by metric learning, including Large Margin Nearest Neighbor methods.
- Results: The method substantially improves image-to-sentence and sentence-to-image retrieval and obtains convincing improvements over CCA for Flickr30K Entities phrase localization.The paper reports retrieval results on Flickr30K and MSCOCO and phrase-localization improvements on Flickr30K Entities.
2. Deep Structure-Preserving Embedding
The method maps image and sentence features into a shared, L2-normalized space and trains them with bidirectional cross-view ranking plus within-view neighborhood constraints. These constraints preserve semantic neighborhoods while supporting image-text matching, with dataset-specific supervision determining which within-view terms are active.
- Embedding architecture: Image and sentence vectors are mapped into a common-dimensional embedding space where similarity uses the inner product, equivalently Euclidean distance after L2 normalization.The two views may initially have different feature dimensions.
- Embedding architecture: The two-branch network uses fully connected layers, ReLU nonlinearities, batch normalization, and final L2 normalization.Each branch can use a different number of layers and can be trained with upstream feature networks.
- Training objective: The stochastic margin-based objective combines bidirectional cross-view ranking constraints with within-view structure-preserving constraints.Hinge loss converts the constraints into the training objective.
- Training objective: For each image, matching sentences must be closer than non-matching sentences by margin m, with an analogous constraint ranking matching images for each sentence.The positive and negative sets are defined separately for image-to-sentence and sentence-to-image retrieval.
- Structure-preserving constraints: Structure-preserving constraints require same-meaning neighbors within each view to be separated from points outside the neighborhood by margin m.Figure 2 illustrates how neighborhood structure can resolve ambiguous cross-view matches for a new image query.
- Optimization: Triplets are sampled within mini-batches using top-K violated matches, while constrained batches add an additional distinct positive sentence for each image.Random pair sampling alone may fail to produce the positive pairs needed for non-empty structure-preserving triplets.
3. Experiments
Experiments evaluate the embedding on bidirectional image-sentence retrieval and phrase localization, while testing nonlinear mappings, bidirectional ranking, structure preservation, and feature choices. The method improves retrieval across Flickr30K and MSCOCO and improves phrase localization over CCA after hard-negative fine-tuning.
- Evaluation: Experiments cover Flickr30K and MSCOCO bidirectional retrieval plus phrase localization on Flickr30K Entities.Flickr30K retrieval uses 1000 test images and 5000 sentences; MSCOCO retrieval uses 1000 test images.
- Ablations: About 4% accuracy improvement comes from replacing linear with nonlinear embeddings across retrieval settings.The comparison is reported in Table 1(b).
- Feature robustness: Structure preservation improves results even with weaker mean-word2vec and tf-idf text features, so highly nonlinear input features are not required.Mean-word2vec results exceed comparable CCA results, while tf-idf results are just below the best Fisher-vector results.
- Phrase localization: On Flickr30K Entities, structure terms generally outperform versions without them, while the pretrained deep model is comparable to CCA at Recall@5 and Recall@10 but lower at Recall@1.Hard-negative fine-tuning later improves Recall@1 and mAP by almost 6% over the best positive-only model and makes results considerably better than CCA.
4. Conclusion
The paper proposes a flexible two-branch image-text embedding trained with bidirectional ranking and metric-learning-inspired structure preservation. Experiments report substantially improved retrieval on Flickr30K and MSCOCO and convincing gains over CCA for Flickr30K Entities phrase localization.
- Conclusion: The method uses a multilayer two-branch network with bidirectional ranking and structure-preserving terms in a margin-based objective.The architecture can operate on varied visual and textual features.
- Conclusion: Retrieval results on Flickr30K and MSCOCO considerably exceed the state of the art.The conclusion presents this as the principal reported outcome.
- Conclusion: Phrase localization on Flickr30K Entities shows convincing improvements over CCA.This extends evaluation beyond global image-sentence retrieval to region-phrase matching.