Source-linked AI summary
Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
Yan Huang, Wei Wang, Liang Wang
TL;DR
Image-sentence matching requires measuring global similarity formed from multiple local object-word relationships. The paper proposes sm-LSTM to select and aggregate salient instance pairs, achieving state-of-the-art results on two public benchmark datasets.
Problem
Accurate image-sentence similarity is difficult because global similarity arises from complex local relationships, while global matching overlooks instances and indiscriminate local matching includes redundant pairs.
Method
sm-LSTM uses multimodal context-modulated attention to select salient image-sentence instance pairs, measure their local similarities, and sequentially aggregate them into a matching score.
Results
The model achieves state-of-the-art results on image annotation and retrieval across two public datasets.
Takeaways & Limitations
Dynamically selecting image-sentence instances and aggregating their similarities is effective for cross-modal retrieval.
Abstract
from arXiv · showhide
Effective image and sentence matching depends on how to well measure their global visual-semantic similarity. Based on the observation that such a global similarity arises from a complex aggregation of multiple local similarities between pairwise instances of image (objects) and sentence (words), we propose a selective multimodal Long Short-Term Memory network (sm-LSTM) for instance-aware image and sentence matching. The sm-LSTM includes a multimodal context-modulated attention scheme at each timestep that can selectively attend to a pair of instances of image and sentence, by predicting pairwise instance-aware saliency maps for image and sentence. For selected pairwise instances, their representations are obtained based on the predicted saliency maps, and then compared to measure their local similarity. By similarly measuring multiple local similarities within a few timesteps, the sm-LSTM sequentially aggregates them with hidden states to obtain a final matching score as the desired global similarity. Extensive experiments show that our model can well match image and sentence with complex content, and achieve the state-of-the-art results on two public benchmark datasets.
1. Introduction
Image-sentence matching must estimate global similarity from complex local relationships, motivating a model that selectively identifies and aggregates salient instance pairs.
- Image-sentence matching supports applications including image annotation, caption querying, and image search.
- Existing methods divide into one-to-one matching using global representations and many-to-many matching using local instance comparisons.
- One-to-one methods overlook that global similarity arises from aggregating local similarities between image objects and sentence words.
- Many-to-many methods compare all instance pairs, but redundant or irrelevant pairs can add noise and degrade the final matching performance.
- The sm-LSTM recurrently selects salient image-sentence instance pairs using multimodal context-modulated attention and aggregates their local similarities across timesteps.It predicts pairwise saliency maps for image regions and sentence words, then forms representations for attended pairs.
2. Related Work
Prior work developed global and local image-sentence matching approaches, while attention-based models focused on related selection or generation tasks rather than the proposed matching objective.
- Global matching methods learn image and sentence representations and associate them with structured or correlation-based objectives.
- Karpathy et al. introduced local similarity learning between image and sentence fragments, while later work collected region-to-phrase correspondences.
- Region- and phrase-based methods use all pairwise instances indiscriminately, although many pairs may be irrelevant to similarity measurement.
- Image-captioning models can compare generated and ground-truth sentences word by word, but they target grammatical sentence generation rather than salient instance selection.
- Related attention models attend to image regions, source-sentence parts, or salient objects for recognition, translation, and caption generation.
3. Selective Multimodal LSTM
The sm-LSTM extracts image and sentence instance candidates, predicts pairwise saliency using multimodal context, and selectively aggregates local similarities into a global matching score.
- Instance candidate extraction: The model extracts sentence words with a BLSTM and image-region candidates from CNN feature maps without additional object detectors.Image regions are evenly divided, while word representations concatenate forward and backward BLSTM states.
- Instance-aware saliency map prediction: Because split words and fixed regions may be irrelevant or incomplete, the model predicts instance-aware saliency maps to highlight useful candidates.Desired instances can combine multiple candidates, such as an object spanning several image regions.
- Global context as reference information: At each timestep, global context identifies expected instances while previous hidden context discourages repeatedly attending to instances already selected.The attention procedure is implemented with a three-way MLP; global context additively modulates initial saliency and previous context subtractively modulates it.
- Global context as reference information: Experiments without global context found that instances such as man and dog were not well attended, supporting its role as reference information.Without global context, attention relies on local initial saliency maps containing little instance or inter-instance relationship information.
- Similarity measurement and aggregation: Saliency-weighted sums form attended image and sentence representations, whose comparison produces a local similarity representation at each timestep.Candidates with higher saliency values contribute more to the attended representations, which are jointly processed by a two-way MLP.
- Similarity measurement and aggregation: An LSTM sequentially aggregates local similarity representations across timesteps, and its hidden state is used to produce the final matching score.A structured objective favors matched pairs over mismatched pairs and penalizes repeatedly attending to the same instance, encouraging broader attention.
4. Experimental Results
Experiments on Flickr30k and Microsoft COCO evaluate sm-LSTM variants for image annotation and retrieval. Results show benefits from selective attention, global-context modulation, sequential timestep choice, and regularization.
- Experimental Setup: Experiments evaluate image annotation and retrieval on Flickr30k and Microsoft COCO using recall-at-k, median rank, and a combined Sum criterion.The datasets contain images paired with human-annotated sentences, and Sum adds R@1, R@5, and R@10 across both tasks.
- Experimental Setup: The full sm-LSTM combines attention and global context, while ablations isolate mean representations, attention, and context effects.The study also evaluates an ensemble of four sm-LSTM variants.
- Comparison with State-of-the-art Methods: sm-LSTM* performs much better than compared methods on both datasets, while single-model sm-LSTM beats DSPE+FV† for image annotation but is slightly worse for image retrieval.Unlike DSPE+FV†, sm-LSTM learns sentence features from scratch in an end-to-end manner.
- Ablation Analysis: Attention improves performance over mean-vector representations, context-only modeling performs much worse than the full model, and global context improves all evaluation criteria over attention alone.The ensemble of four variants further improves performance.
- Number of Timesteps: Three timesteps give the best performance; increasing T beyond 3 slightly reduces results because the more complex network can overfit by attending to redundant instances.The model is described as capturing local similarities by iteratively visiting image and sentence instances three times.
- Regularization Analysis: Regularization improves performance when λ>0, with λ=100 producing the largest improvement, especially for image annotation.The regularizer is introduced because the model may repeatedly focus on the same informative instance instead of covering the entire content.
- Attention Visualization: Saliency visualizations show attention shifting across semantically corresponding image regions and sentence words over three timesteps.Examples include sequential focus on “giraffe,” “children,” and “park,” with matching image regions; averaged maps emphasize central regions first and surrounding or lower regions later.
5. Conclusions
The paper proposes sm-LSTM for instance-aware image–sentence matching and reports state-of-the-art results on image annotation and retrieval. It combines multimodal context-modulated attention with multimodal LSTM-based local similarity measurement and aggregation.
- sm-LSTM selects salient pairwise instances from images and sentences using multimodal context-modulated attention.
- A multimodal LSTM measures and aggregates local similarities between selected image–sentence instances.
- The model systematically studies global context modulation in attention and reports significant performance improvement.
- The model achieves state-of-the-art results on image annotation and retrieval tasks.
- Future work includes more advanced context modulation, evaluation on more datasets, and joint fine-tuning of the pretrained CNN.