Source-linked AI summary
Stacked Cross Attention for Image-Text Matching
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, Xiaodong He
TL;DR
Image-text matching requires recovering latent semantic alignments between image regions and sentence words, while prior approaches either aggregate pair similarities or capture limited alignments. Stacked Cross Attention uses each modality as context for the other to infer full alignments and image-text similarity. SCAN achieves state-of-the-art results on Flickr30K and MS-COCO, with reported relative Recall@1 gains across retrieval tasks.
Problem
Image-text matching lacks differential modeling of important region-word pairs or captures only limited semantic alignments, despite the importance of latent correspondence for interpretability.
Method
Stacked Cross Attention attends to words with respect to image regions and to image regions with respect to words, using both modalities as context to infer similarity.
Results
SCAN achieves state-of-the-art results on Flickr30K and MS-COCO, improving Recall@1 by 22.1% and 18.2% on Flickr30K and 17.8% and 16.6% on MS-COCO 5K.
Takeaways & Limitations
The learned Stacked Cross Attention can expose fuller visual-semantic alignments and make vision-language models more interpretable.
Abstract
from arXiv · showhide
In this paper, we study the problem of image-text matching. Inferring the latent semantic alignment between objects or other salient stuff (e.g. snow, sky, lawn) and the corresponding words in sentences allows to capture fine-grained interplay between vision and language, and makes image-text matching more interpretable. Prior work either simply aggregates the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or uses a multi-step attentional process to capture limited number of semantic alignments which is less interpretable. In this paper, we present Stacked Cross Attention to discover the full latent alignments using both image regions and words in a sentence as context and infer image-text similarity. Our approach achieves the state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K, our approach outperforms the current best methods by 22.1% relatively in text retrieval from image query, and 18.2% relatively in image retrieval with text query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1 using the 5K test set). Code has been made available at: https://github.com/kuanghuei/SCAN.
1 Introduction
Image-text matching benefits from recovering latent correspondences between salient image regions and sentence words. Stacked Cross Attention attends to regions and words as mutual context, discovers alignments simultaneously, and achieves state-of-the-art retrieval results.
- Image-text matching supports cross-modal retrieval between images and sentences with visual descriptions.
- Latent correspondences link sentence words to salient image regions, attributes, and actions, making fine-grained matching more interpretable.
- Stacked Cross Attention differentially attends to words and image regions using each modality as context when inferring image-text similarity.
- Unlike fixed-step attentional models, Stacked Cross Attention discovers all possible semantic alignments simultaneously.This is intended to provide more comprehensive correspondence inference when the number of alignments varies across image-sentence pairs.
- 22.1% relative improvement in Flickr30K text retrieval from image query and 18.2% in image retrieval with text query are reported at Recall@1.
- 17.8% relative improvement in MS-COCO sentence retrieval and 16.6% in image retrieval are reported at Recall@1 on the 5K test set.
2 Related Work
Prior image-text matching research commonly maps whole images and sentences into a shared semantic space, while attention-based work adaptively focuses on visual or linguistic content. These lines of work motivate region-word correspondence modeling.
- Many prior methods map whole images and full sentences into a common semantic vector space for image-text matching.
- Earlier cross-view models used CNN image encoders, RNN sentence encoders, and hinge-based triplet ranking objectives.
- Hard negatives, generative objectives, and attention mechanisms represent major directions in prior image-text matching research.
- Unlike the proposed method, whole-image and full-sentence embedding approaches do not model latent correspondence between image regions and words.
3 Learning Alignments with Stacked Cross Attention
SCAN maps image regions and contextualized words into a common space, then uses two complementary cross-attention directions to infer image-sentence similarity. Its alignment objective trains matching scores against hard negatives using a triplet loss.
- Inputs and representations: SCAN represents an image as region features and a sentence as contextual word features in a shared embedding space.Image regions are detected at object or salient-stuff level, while an RNN embeds words together with sentence context.
- Image-Text Stacked Cross Attention: Image-Text attention first attends to words for each image region, then compares each region with its attended sentence vector to estimate region importance.The resulting region-level relevance scores are pooled into an image-sentence similarity score.
- Similarity pooling: LogSumExp pooling emphasizes the most relevant region-word correspondences, while average pooling provides an alternative similarity aggregation.The pooling choice controls how local relevance scores are summarized into the final image-text similarity.
- Text-Image Stacked Cross Attention: Text-Image attention reverses the direction by attending to image regions for each word, then comparing each word with its attended image vector.This complementary formulation estimates word-level relevance before pooling the relevance scores.
- Alignment objective: The model learns alignments with a hinge-based triplet objective that compares positive image-sentence pairs against negative sentences and images.Training commonly focuses on the hardest negatives in each stochastic-gradient mini-batch for computational efficiency.
- Image representation: Bottom-up attention uses Faster R-CNN features to represent salient image regions, including objects and difficult-to-localize stuff such as sky, grass, and buildings.Each selected region is transformed into an embedding vector, yielding a set-based image representation.
4 Experiments
Experiments evaluate SCAN on Flickr30K and MS-COCO using cross-modal retrieval and ablations. SCAN achieves strong retrieval results, while ablations examine latent alignment, pooling, hard negatives, normalization, and sentence encoding.
- Experimental Setup: SCAN is evaluated on MS-COCO and Flickr30K using sentence retrieval and image retrieval measured by Recall@K.The evaluation includes comparisons with prior architectures and ablation studies.
- Results on Flickr30K: 67.9 R@1 is achieved for sentence retrieval on Flickr30K by SCAN i-t AVG, a 22.1% relative improvement over DPC.The result is reported for image queries retrieving sentences.
- Results on Flickr30K: 48.6 R@1 is achieved for image retrieval on Flickr30K by an ensemble of t-i AVG and i-t LSE, an 18.2% relative improvement over SCO.The ensemble averages predicted similarity scores and is selected on the validation set.
- Results on MS-COCO: 17.8% relative improvement in sentence retrieval and 16.6% in image retrieval are obtained by SCAN t-i AVG + i-t LSE on the MS-COCO 5K test set.Both the best single model and validation-selected ensemble outperform SCO on all metrics in this setting.
- Ablation Studies: Inferring region-word alignments improves over whole-image and whole-sentence matching, while SCAN further improves over Sum-Max models.Sum-Max aggregates all region-word similarity scores, whereas VSE++ uses single image and sentence embeddings.
- Ablation Studies: 48.2% improvement in sentence retrieval R@1 comes from hard negatives; bidirectional GRU adds 4.3 sentence-retrieval points and 0.7 image-retrieval points.SUM and MAX pooling are weaker than average or LogSumExp pooling, and image-embedding normalization has little effect.
5 Visualization and Analysis
The visualization analysis examines SCAN’s learned attention over image regions for individual sentence words. The examples show focused attention and shifts among objects, attributes, and actions, alongside qualitative retrieval results.
- Visualizing Attention: The visualization outlines the maximum-attention region in red and uses brightness to indicate attention strength.The attention reflects the estimated importance of both the image region and the word.
- Visualizing Attention: SCAN visualizations show attention weights for each word in a sentence description across the corresponding image regions.Regional brightness represents attention strength based on both region and word importance.
- Visualizing Attention: Words such as “boy”, “holding”, “tennis”, and “racket” receive strong, focused attention in the example visualization.The displayed sentence is “A young boy is holding a tennis racket.”
- Qualitative Retrieval: Qualitative retrieval figures present top-5 sentences for image queries and top-3 images for sentence queries, with true matches outlined in green.Retrieved items are ranked by the similarity scores predicted by SCAN.
6 Conclusions
SCAN achieves state-of-the-art performance on Flickr30K and MS-COCO and supports interpretable image-text matching through learned attention. Qualitative retrieval examples show strong top-ranked matches, while some errors remain in complex scenes.
- SCAN gives state-of-the-art performance on Flickr30K and MS-COCO in all reported measures.
- Qualitative image-query examples retrieve correct sentences among the top-ranked results, including for complex and cluttered scenes.Some reasonable mismatches and errors are also observed, with poor action detection suggested as one possible cause.
- Sentence-query examples retrieve the ground-truth image in the top-three results, while other returned images can also be reasonable.
- The learned attention can make vision-language models more interpretable by exposing latent correspondences between image regions and words.
Appendix Overview
The supplementary material covers training details, an ROI-count ablation, and additional qualitative retrieval and attention examples.
- Section A presents details of training the proposed Stacked Cross Attention Network.
- Section B presents an ablation study on the number of Region of Interests (ROIs).
- Section C presents qualitative examples of attended image regions and both directions of image-text retrieval.
A Details of Training
Training uses Adam with dataset-specific learning-rate schedules, triplet-loss margin α = 0.2, mini-batches of 128, and gradient clipping at 2.0.
- Adam trains Flickr30K models for 15 epochs at 0.0002 followed by 15 epochs at 0.00002.
- Adam trains MS-COCO models for 10 epochs at 0.0005 followed by 10 epochs at 0.00005.
- The triplet-loss margin is α = 0.2, mini-batch size is 128, and the maximum gradient norm is clipped at 2.0.
B Details of Bottom-up Attention
Bottom-up attention extracts image ROIs with Faster R-CNN, while an ablation identifies k = 36 as the preferred ROI count under matched training and inference settings.
- Faster R-CNN with ResNet-101 extracts image ROIs, using IoU thresholds of 0.7 for region-proposal suppression and 0.3 for object-class suppression.
- When training and inference use the same ROI count, k = 36 yields the best results.Performance drops at k = 48 and k = 60 are suspected to result from noisy low-ranking regions.
- Using 12 or 24 regions at inference with a model trained on 36 produces similar drops to using those smaller counts during both training and inference.
- The ROI ablation reports Recall@K and average times for similarity computation, image encoding, and text encoding.
C Additional Examples
Additional qualitative examples illustrate how the Text-Image Stacked Cross Attention Network aligns words with image regions and ranks retrieval results. The examples also expose reasonable mismatches and failure cases across Flickr30K and MS-COCO.
- Image-text matching: The model assigns attention to image regions corresponding to words for objects, stuff, attributes, and actions.Examples include “bike,” “dog,” “sidewalk,” “building,” “red,” “person,” “rides,” “pier,” and “sunset.”
- Image-text matching: Attention can cover multiple objects for a word such as “family” and fine details such as a polar bear’s leg for “standing.”
- Sentence retrieval: Additional sentence-retrieval examples for image queries are shown separately for Flickr30K and MS-COCO.The figures mark incorrect results with red indicators; reasonable mismatches remain marked as incorrect.
- Image retrieval: Additional image-retrieval examples for sentence queries display the top five ranked images, with true matches outlined in green and false matches in red.Each sentence corresponds to one ground-truth image in the Flickr30K and MS-COCO examples.
- Failure cases: The qualitative retrieval examples include reasonable mismatches and failures attributed to attention behavior or visual confusion.One Flickr30K case ranks two reasonable mismatches before the ground truth, while an MS-COCO mismatch may confuse plastic and glass bakery cases.