Source-linked AI summary
Step-Wise Hierarchical Alignment Network for Image-Text Matching
Zhong Ji, Kexin Chen, Haoran Wang
TL;DR
Image-text matching remains challenging because machines must infer cross-modal similarity, particularly for hard negatives with similar semantics and subtle contextual differences. SHAN progressively aligns image regions with words and then aligns global context with local and global representations. The paper reports state-of-the-art performance on two benchmark datasets.
Problem
Image-text matching requires machines to infer cross-modal similarity, especially for hard negatives with similar semantic content but subtle contextual differences.
Source: §1 · pp. 1–2Method
SHAN progressively performs region-word local-to-local alignment, followed by context-level global-to-local and global-to-global alignment.
Source: §1 · p. 2Results
SHAN achieves state-of-the-art performance on Flickr30k and MS-COCO, with maximal text-retrieval R@1 improvements of 2.8% and 1.3%, respectively.
Source: §1 · p. 2, §4.3 · p. 5Takeaways & Limitations
The reported visualizations show word-to-region matches at the first alignment step and highlight image regions relevant to global context at the second.
Source: §4.5 · p. 6
Abstract
from arXiv · showhide
Image-text matching plays a central role in bridging the semantic gap between vision and language. The key point to achieve precise visual-semantic alignment lies in capturing the fine-grained cross-modal correspondence between image and text. Most previous methods rely on single-step reasoning to discover the visual-semantic interactions, which lacks the ability of exploiting the multi-level information to locate the hierarchical fine-grained relevance. Different from them, in this work, we propose a step-wise hierarchical alignment network (SHAN) that decomposes image-text matching into multi-step cross-modal reasoning process. Specifically, we first achieve local-to-local alignment at fragment level, following by performing global-to-local and global-to-global alignment at context level sequentially. This progressive alignment strategy supplies our model with more complementary and sufficient semantic clues to understand the hierarchical correlations between image and text. The experimental results on two benchmark datasets demonstrate the superiority of our proposed method.
1 Introduction
Image-text matching remains difficult because machines must infer cross-modal similarity, especially for hard negatives with similar semantics but subtle differences. SHAN progressively combines fragment- and context-level alignment, and the paper reports state-of-the-art performance on Flickr30k and MS-COCO.
1 Introduction
- Earlier global-embedding methods overlook local fragments, while fine-grained interaction methods align image regions and text words.Global representations may miss fine-grained details and include noisy background information.Source: §1 · pp. 1–2
- Hard negatives can share salient foreground entities while differing in contextual attributes or relationships, challenging methods that rely on single-step, fragment-level inference.The example contrasts “people,” “surfboard,” and “wave” with contextual clues such as “two” and “near each other.”Source: §1 · p. 2
- SHAN progressively performs local-to-local fragment alignment, then global-to-local and global-to-global context alignment.The method uses cross-attention for region-word alignment and fuses intermediate information to form global context representations.Source: §1 · p. 2
- The paper reports state-of-the-art performance for SHAN on both Flickr30k and MS-COCO.Detailed visualizations of L2L and G2L alignment are also reported to qualitatively verify the method.Source: §1 · p. 2
2 Related Work
Image-text matching research includes global embedding approaches and fine-grained interaction approaches. SHAN aligns at both fragment and context levels.
2 Related Work
- Global-embedding approaches learn a shared space for image and text representations and compare their similarity directly.Examples include CNN and RNN global representations aligned with a hinge-based triplet ranking loss [Kiros et al., 2014], and hard-negative mining in the ranking loss [Faghri et al., 2017].Source: §2 · p. 2
- Fine-grained interaction methods explore image-text correspondences, including region-word similarity aggregation [Karpathy and Fei-Fei, 2015].Chen et al. [2020] use multiple alignment steps but pay less attention to context-level information.Source: §2 · p. 2
- SHAN performs visual-semantic alignment from both fragment-level and context-level perspectives.This contrasts with the context-level information not emphasized by Chen et al. [2020].Source: §2 · p. 2
3 Our Method
SHAN progressively explores semantic alignments between image and text. Its framework comprises feature representation, hierarchical alignment, and an objective function for model learning.
3 Our Method
- SHAN progressively explores semantic alignments between image and text.The framework is illustrated in Figure 2(a).Source: §3 · p. 2
- The method's framework includes feature representations for images and text and the implementation of the SHAN alignment model.Source: §3 · p. 2
- The framework also includes an objective function for model learning.Source: §3 · p. 2
3.1 Feature Representation
Image features are extracted from salient regions detected by a pretrained Faster R-CNN, while text features are built from word representations and contextualized with a bidirectional GRU.
Image Representation
- Each image is represented using 36 salient regions detected by Faster R-CNN pretrained on Visual Genome, then projected into D-dimensional vectors.Source: Image Representation · pp. 2–3
Text Representation
- Text word representations concatenate pretrained GloVe embeddings with learnable embeddings, then a bidirectional GRU contextualizes them by averaging its forward and backward hidden states.Source: Text Representation · p. 3
3.2 Step-wise Hierarchical Alignment
SHAN performs image-text matching through three progressive alignment steps: fragment-level local-to-local, then context-level global-to-local and global-to-global alignment. These stages combine cross-modal fragment interactions with contextual representations to score image-text relevance.
Fragment-level L2L Alignment
- Bidirectional cross-attention aligns image regions and text words, then averages region-based and word-based relevance scores, balanced by µ1, for the L2L score.The region-word affinity matrix encodes semantic similarity between each region and word.Source: Fragment-level L2L Alignment · p. 3
Context-level G2L Alignment
- Adaptive fusion combines each region or word feature with attended information from the other modality; self-attention pools fused features into global image and text representations for bidirectional G2L alignment.The gate balances the original feature and attended cross-modal information; the G2L score sums the visual- and textual-global-related matching scores.Source: Context-level G2L Alignment · p. 4
Context-level G2G Alignment
- G2G alignment fuses the intermediate context representations with cross-modal attended information, then computes their cosine-distance matching score.The proposed fusion is intended to encourage mutual-information propagation and learn more discriminative context-level features.Source: Context-level G2G Alignment · p. 4
3.3 Learning Objective
The model sums three alignment-level similarities and trains with a bidirectional hinge-based triplet ranking loss that favors matched pairs over unmatched ones.
3.3 Learning Objective
- The final image-text similarity sums the L2L, G2L, and G2G matching scores.These scores correspond to local-to-local, global-to-local, and global-to-global alignment.Source: §3.3 · p. 4
- A bidirectional hinge-based triplet ranking loss trains the model to score matched image-text pairs above unmatched pairs.The loss uses contrastive negative samples from the mini-batch and a tuning margin parameter.Source: §3.3 · pp. 4–5
4 Experiments
Experiments evaluate SHAN on Flickr30k and MS-COCO using text and image retrieval. SHAN achieves the best retrieval results on both datasets, while ablations and attention visualizations examine its alignment components.
Dataset
- Flickr30k has 31,783 images with five sentences each, while MS-COCO has 123,287 images with five descriptions each.The experiments use the stated train, validation, and test splits for each dataset.Source: Dataset · p. 5
Evaluation Metric
- Evaluation covers text and image retrieval using Recall@K for K = 1, 5, 10 and the summed metric Rsum.R@K measures the fraction of ground-truth instances retrieved in the top K results.Source: Evaluation Metric · p. 5
4.2 Implementation Details
- The study compares text-to-image-only and image-to-text-only SHAN variants with SHAN-full, which applies hierarchical alignment in both directions.Models are trained for 30 epochs on each dataset.Source: §4.2 · p. 5
4.3 Comparison with State-of-the-art Methods
- SHAN achieves the best retrieval results on Flickr30k and MS-COCO; text-retrieval R@1 gains reach 2.8% and 1.3%, respectively.For image retrieval, its R@1 is 55.3% on Flickr30k and 62.6% on MS-COCO.Source: §4.3 · p. 5
4.4 Ablation Studies
- Ablations report that combining pretrained Glove with learnable embeddings performs better than either embedding choice alone, and both G2L and G2G modules provide performance boosts.The ablations remove the G2L and G2G modules in turn.Source: §4.4 · p. 6, Table 3 · p. 6
4.5 Visualization
- L2L attention matches words to corresponding image regions, while G2L attention highlights meaningful regions for global context and suppresses trivial ones.The figure visualizes attended visual information progressively across the two alignment steps.Source: §4.5 · p. 6, Figure 3 · p. 6
5 Conclusion
SHAN progressively performs fragment-level L2L alignment, context-level G2L alignment, and context-level G2G alignment. Experiments and ablations on two benchmarks show competitive results.
5 Conclusion
- SHAN progressively combines fragment-level L2L, context-level G2L, and context-level G2G alignment to learn visual-semantic correspondence.Experiments and ablation studies on two widely used benchmark datasets demonstrate competitive results.Source: §5 · p. 6