Source-linked AI summary
IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, Jungong Han
TL;DR
Existing image-text retrieval methods often align diverse semantics uniformly, making sophisticated vision-language correspondences difficult to capture. IMRAM progressively performs multiple alignments with recurrent attention memory, and reports state-of-the-art or superior performance across benchmark and practical advertisement datasets.
Problem
Existing attention-based retrieval methods treat diverse semantic concepts uniformly, although image-text semantics can have complex correlations and latent structure.
Method
IMRAM progressively aligns image and text fragments through iterative cross-modal attention while distilling alignment knowledge from earlier steps to later ones.
Results
IMRAM consistently outperforms compared methods on Flickr8K, Flickr30K, MS COCO, and KWAI-AD, achieving state-of-the-art performance on the evaluated datasets.
Takeaways & Limitations
The results support IMRAM's effectiveness for cross-modal image-text retrieval in benchmark and practical business advertisement scenarios.
Abstract
from arXiv · showhide
Enabling bi-directional retrieval of images and texts is important for understanding the correspondence between vision and language. Existing methods leverage the attention mechanism to explore such correspondence in a fine-grained manner. However, most of them consider all semantics equally and thus align them uniformly, regardless of their diverse complexities. In fact, semantics are diverse (i.e. involving different kinds of semantic concepts), and humans usually follow a latent structure to combine them into understandable languages. It may be difficult to optimally capture such sophisticated correspondences in existing methods. In this paper, to address such a deficiency, we propose an Iterative Matching with Recurrent Attention Memory (IMRAM) method, in which correspondences between images and texts are captured with multiple steps of alignments. Specifically, we introduce an iterative matching scheme to explore such fine-grained correspondence progressively. A memory distillation unit is used to refine alignment knowledge from early steps to later ones. Experiment results on three benchmark datasets, i.e. Flickr8K, Flickr30K, and MS COCO, show that our IMRAM achieves state-of-the-art performance, well demonstrating its effectiveness. Experiments on a practical business advertisement dataset, named \Ads{}, further validates the applicability of our method in practical scenarios.
1. Introduction
IMRAM addresses the difficulty of aligning diverse, structured image-text semantics by progressively refining cross-modal correspondences. It outperforms compared methods on benchmark and advertisement datasets.
- Cross-modal image-text retrieval requires understanding both modality content and their inter-modal correspondence.
- Existing methods align diverse semantic concepts uniformly, despite correlations among objects, attributes, and relations and their latent structural organization.
- Human comparison progressively associates objects before mining attributes and relationships through revisiting images and texts.
- IMRAM combines iterative cross-modal attention with memory distillation to refine alignment knowledge from early matching steps to later ones.
- IMRAM outperforms compared methods across Flickr8K, Flickr30K, MS COCO, and the KWAI-AD business advertisement dataset.The benchmark and practical-dataset evaluations are reported as evidence of effectiveness across these settings.
2. Related work
Related work includes coarse- and fine-grained image-text matching, attention-based alignment, sequential matching, and externally resourced approaches. IMRAM differs by recurrently transferring fragment-alignment knowledge across iterative matching steps.
- Coarse-grained methods map whole images and texts into a common embedding space, while fine-grained methods align image and text fragments.
- Attention-based methods use fragment-level alignments to discover fine-grained vision-language correspondence.
- IMRAM refines fragment alignment iteratively and distills matched-semantic knowledge after each matching step.
- Unlike sequential methods using modality-specific context, IMRAM transfers fragment-alignment knowledge to successive steps through recurrent attention memory.
- Some recent approaches use large-scale external image-text resources or pretrained cross-modal BERT models to improve performance.
3. Methodology
IMRAM represents images with region features and texts with contextual word features, then progressively refines their correspondence through recurrent attention memory. Iterative matching aggregates step-wise scores and trains the model using hard-negative triplet ranking.
- Cross-modal Feature Representation: IMRAM extracts region-level image features with a pretrained CNN, projects them into d-dimensional vectors, and normalizes the resulting representation.An image is represented as V = {v_i | i = 1, ..., m}, where m is the number of detected regions.
- Cross-modal Feature Representation: Texts are encoded at word level using word embeddings and a bidirectional GRU that summarizes forward and backward context.The resulting word features share the same dimensionality as image-region features and are normalized.
- RAM: Recurrent Attention Memory: The recurrent attention memory aligns two feature sets with cross-modal attention, then distills alignment information to refine query features for subsequent matching.The RAM block accepts image and text feature sets in either query-response direction and combines each query feature with its corresponding alignment feature.
- RAM: Recurrent Attention Memory: A modified gating mechanism fuses each query feature with shared cross-modal semantics, filtering trivial information while retaining query-specific information.The gate selects salient information, and the resulting context features implicitly involve and recalibrate relationships within the response modality.
- Iterative Matching with Recurrent Attention Memory: IMRAM applies RAM iteratively to step-wise image and text features, derives region-based and word-based matching scores at each step, and sums them after K steps.The method uses two independent RAM blocks for image-grounded and text-grounded iterative matching.
- Loss Function: The model is trained end-to-end with a triplet-wise ranking objective that compares matched pairs against the closest hard negatives within each minibatch.The loss includes matching scores from every iterative step, directly supervising correspondence learning at each step.
4. Experiment
Experiments evaluate IMRAM on three benchmark datasets and the KWAI-AD advertisement dataset, using retrieval metrics and model analyses. Results show consistent gains, while analyses indicate that iterative matching progressively refines attention and captures different semantic types at different steps.
- Datasets and evaluation: IMRAM is evaluated on Flickr8K, Flickr30K, MS COCO, and the newly collected KWAI-AD advertisement dataset.KWAI-AD contains 81,653 image-text pairs with weak associations and affective rather than factual semantics.
- Datasets and evaluation: Recall at K measures both text retrieval and image retrieval, with R@1, R@5, R@10, and aggregate R@sum reported.R@sum is the summation of all evaluation metrics.
- Results on three benchmark datasets: 12.6%, 19.2%, 8.7% and 5.6% are Full-IMRAM’s R@sum improvements over SCAN* on Flickr8K, Flickr30K, MS COCO (1K) and MS COCO (5K), respectively.The passage also reports gains across text- and image-retrieval recall metrics.
- Model analysis: K = 2 and K = 3 consistently outperform K = 1 across IMRAM variants, while Full-IMRAM outperforms Image-IMRAM and Text-IMRAM.The analysis is reported on MS COCO (5K).
- Qualitative analysis: Attention is refined across matching steps and gradually focuses on regions corresponding to matched semantic concepts.This qualitative analysis uses Text-IMRAM with K = 3.
- Qualitative analysis: Nouns dominate at the first matching step, while verbs and adjectives contribute more during subsequent steps.The observation comes from statistical analysis of Text-IMRAM on MS COCO with K = 3.
- Results on the newly collected Ads dataset: On KWAI-AD, overall performance is lower than on benchmark datasets, yet IMRAM obtains substantial improvements over compared models.The results indicate both the challenge of real-world advertisement retrieval and the method’s effectiveness in that setting.
5. Conclusion
IMRAM addresses semantic complexity in cross-modal image-text retrieval by progressively refining correspondence with iterative matching and recurrent attention memory. It achieves state-of-the-art performance across three benchmarks and the KWAI-AD dataset.
- IMRAM handles complex semantics through iterative matching and recurrent attention memory.It aligns fragments progressively and refines alignment knowledge from early steps to later ones.
- IMRAM is evaluated on Flickr8K, Flickr30K, MS COCO, and the KWAI-AD business advertisement dataset.
- IMRAM consistently outperforms compared methods and achieves state-of-the-art performance on all evaluated datasets.