Source-linked AI summary
Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching
Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzhu Zhang, Bin Wang, Yongdong Zhang
TL;DR
Image-text matching must identify shared semantics without irrelevant fragments causing semantic misalignment. BFAN removes irrelevant fragments through bidirectional focal attention and significantly outperforms state-of-the-art methods on Flickr30K and MSCOCO.
Problem
Existing attention methods combine all image or text fragments, allowing irrelevant fragments to disturb shared semantics and cause semantic misalignment.
Method
BFAN preassigns attention, identifies relevant fragments using relative attention, and reassigns attention in both image-to-text and text-to-image directions.
Results
BFAN significantly outperforms state-of-the-art methods on the Flickr30K and MSCOCO image-text matching benchmarks.
Takeaways & Limitations
Bidirectional focal attention avoids preference for long text or complex images while concentrating shared semantics on relevant fragments.
Abstract
from arXiv · showhide
Learning semantic correspondence between image and text is significant as it bridges the semantic gap between vision and language. The key challenge is to accurately find and correlate shared semantics in image and text. Most existing methods achieve this goal by representing the shared semantic as a weighted combination of all the fragments (image regions or text words), where fragments relevant to the shared semantic obtain more attention, otherwise less. However, despite relevant ones contribute more to the shared semantic, irrelevant ones will more or less disturb it, and thus will lead to semantic misalignment in the correlation phase. To address this issue, we present a novel Bidirectional Focal Attention Network (BFAN), which not only allows to attend to relevant fragments but also diverts all the attention into these relevant fragments to concentrate on them. The main difference with existing works is they mostly focus on learning attention weight while our BFAN focus on eliminating irrelevant fragments from the shared semantic. The focal attention is achieved by pre-assigning attention based on inter-modality relation, identifying relevant fragments based on intra-modality relation and reassigning attention. Furthermore, the focal attention is jointly applied in both image-to-text and text-to-image directions, which enables to avoid preference to long text or complex image. Experiments show our simple but effective framework significantly outperforms state-of-the-art, with relative Recall@1 gains of 2.2% on both Flicr30K and MSCOCO benchmarks.
1 INTRODUCTION
Image-text matching seeks shared semantics across vision and language, but conventional attention still aggregates irrelevant fragments that disturb semantic alignment. BFAN addresses this by eliminating irrelevant fragments and applying focal attention bidirectionally.
- Image-text matching aims to find and associate shared semantics between images and text to bridge the vision-language semantic gap.
- Attention-based methods emphasize important fragments, but they still aggregate irrelevant regions or words into shared semantics.
- BFAN focuses attention only on relevant fragments, addressing semantic misalignment caused by irrelevant-fragment interference.
- BFAN preassigns attention, identifies relevant fragments using relative attention, and reassigns attention to concentrate on them.
- The unified bidirectional framework applies focal attention in image-to-text and text-to-image directions to avoid preference for long text or complex images.
2 RELATED WORK
Image-text matching research includes one-to-one and many-to-many correspondence methods. BFAN extends attention-based many-to-many matching by removing irrelevant fragments and jointly using both retrieval directions.
- One-to-one approaches associate whole images and texts, whereas many-to-many approaches learn correspondence between image regions and text words.
- One-to-one methods commonly project image and text representations into a shared space and optimize their relevance.
- Many-to-many methods learn region-word correspondence, with attention dynamically changing target-fragment importance based on the other modality.
- BFAN preassigns attention, identifies relevant parts through within-modality relationships, and reassigns focal attention in both retrieval directions.
- Conventional attention aggregates all fragments as a weighted combination, allowing irrelevant regions or words to disturb the target semantic.
3 METHOD
BFAN comprises feature extraction, focal attention, and an objective function. Its focal-attention module interactively processes regions and words before the matching objective is optimized.
- The BFAN framework consists of feature extraction, focal attention, and an objective-function component.
- The method analyzes general attention before introducing focal attention for text-to-image and image-to-text matching.
- BFAN includes an objective function and feature-extraction details alongside its focal-attention mechanism.
3.1 General Attention Framework
General image-text matching projects words and regions into a shared space and aggregates local semantics. Because irrelevant fragments disturb these representations, the framework motivates selecting only relevant fragments.
- Each text word and image region is projected into a common d-dimensional space before shared semantics are associated.
- Shared semantics consist of multiple local correspondences between image regions and text words.
- The objective is to maximize the relevance of each selected local shared semantic.
- Existing attention assigns higher weights to relevant fragments but aggregates all fragments into each shared-semantic representation.
- Irrelevant fragments can disturb shared semantics and cause semantic misalignment, motivating representations built from relevant subsets.
3.2 Our Focal Attention
BFAN identifies relevant fragments through relative attention and removes irrelevant fragments before renormalizing attention. It applies this focal selection in both text-to-image and image-to-text directions, then sums directional relevance scores.
- Focal attention: Focal attention learns a scoring function that identifies relevant fragments and removes irrelevant fragments from shared semantics.Fragments with scores greater than zero are treated as relevant.
- Focal attention: Relative fragment importance is determined from comparisons with other fragments and their confidence scores, rather than fixed absolute similarity margins.A fragment is relevant when it is similar to other relevant, high-confidence fragments.
- Text-to-image focal attention: In text-to-image matching, the model preassigns region attention from cosine similarities with words, selects relevant regions, and renormalizes attention over them.Irrelevant regions receive zero attention after selection.
- Image-to-text focal attention: In image-to-text matching, image regions are fixed while semantically relevant words are selected and given reassigned attention.The resulting shared semantic is a weighted combination of the selected words.
- Bidirectional focal attention: The two directional modules compute separate global relevance scores and combine them by summation, allowing both directions to contribute to the final score.The authors state that this overall-score restraint relaxes constraints and avoids overfitting associated with separately restraining each direction.
3.3 Objective Function
BFAN is optimized with a structured ranking loss that separates relevant image-text pairs from hard negative pairs using a fixed margin.
- Ranking objective: The structured ranking loss maximizes relevance for matched image-text pairs while minimizing relevance for hard negative pairs in each mini-batch.Hard negatives are the irrelevant pairs with the maximum relevance score.
- Ablation evaluation: Ablation studies evaluate the contribution of focal attention and bidirectional focal attention on Flickr30K and MSCOCO.The cited table passages identify Tables 3 and 4 as ablation studies on the two benchmarks.
- Ranking objective: The loss contains image-negative and text-negative hinge terms comparing each matched pair against its corresponding hard negative.The margin parameter α is set to 0.2.
3.4 Feature Extraction
The model represents images with salient detected regions and represents text with contextualized word features produced by a bidirectional GRU.
- Image features: Images are represented by the top K=36 salient objects detected with Faster R-CNN and mean-pooled convolutional features from their bounding boxes.The regions are selected according to detector-predicted object scores.
- Text features: Text is split into words, embedded into low-dimensional vectors, and encoded with a bidirectional GRU to integrate forward and backward context.The resulting word representations are d-dimensional, like the image-region features.
4 EXPERIMENTS
Experiments evaluate BFAN on Flickr30K and MSCOCO using bidirectional retrieval metrics and comparisons with established image-text matching baselines. Attention visualizations contrast focal attention with conventional attention.
- Datasets: Flickr30K and MSCOCO provide the evaluation benchmarks, with five texts associated with each image and predefined training, validation, and test splits.MSCOCO results are averaged over five folds on 1K test images.
- Evaluation metrics: Performance is measured with Recall@1 and Recall@5 for image-to-text and text-to-image retrieval, plus directional means and an overall Recall sum.Recall measures the proportion of correct retrieved items among the top K results.
- Implementation: The network is trained with Adam on one NVIDIA TITAN Xp, using benchmark-specific learning rates, batch size 32, and 15 or 20 training epochs.The learning rates are 0.0002 on Flickr30K and 0.0005 on MSCOCO.
- Baselines: The experiments compare BFAN with many-to-many and one-to-one baselines, including SCAN, BSSAN, VSE++, DANs, and GXN.BFAN-prob and BFAN-equal provide alternative focal-attention implementations, with single and ensemble variants.
- Visualization: Attention visualizations compare focal attention with conventional attention by showing brighter regions as receiving more attention and outlining relevant versus irrelevant regions.The figure is intended to show focal attention concentrating on relevant regions while conventional attention also attends to irrelevant ones.
4.2 Comparison Results
BFAN outperforms state-of-the-art image-text matching methods on Flickr30K and MSCOCO, with focal and bidirectional attention both contributing to performance. Its gains are especially pronounced for Recall@1 and remain stable across datasets.
- Comparison Results: 68.1% and 50.8% Recall@1 are achieved by BFAN for image-to-text and text-to-image matching on Flickr30K, respectively.Text-to-image Recall@1 exceeds 50% for the first time on Flickr30K and gains 2.2% relative to SCAN.
- Comparison Results: BFAN outperforms SCO and GXN while learning more fine-grained region-word correspondence.The paper identifies this correspondence as significant for real multimodal applications.
- Comparison Results: 5.3% ∼5.4% relative rsum gains over the state-of-the-art single model are reported on MSCOCO.The ensemble model also outperforms the best ensemble model.
- Ablation Study: Removing focal attention or using it in only one direction reduces matching performance, especially for Recall@1.On Flickr30K, single-direction variants decrease all Recall values by nearly 2% on average compared with the full single model.
- Ablation Study: Focal attention and bidirectional processing complement each other and remain stable across Flickr30K and MSCOCO.On MSCOCO, the full single-direction variants outperform other ablation models across all evaluation metrics.
4.4 Attention Visualization
The visualizations show BFAN concentrating attention on query-relevant regions and improving retrieval compared with the baseline. Its examples emphasize resistance to distractors and finer-grained image-text correspondence.
- Attention Visualization: BFAN aligns query words with relevant image regions, whereas the baseline also attends to irrelevant regions such as sky and grass.The visualization compares brightness-weighted attention over image bounding boxes.
- Text-to-Image Matching: BFAN correctly ranks the target image first despite distractor regions matching individual query keywords.The baseline incorrectly favors an image containing locally matching “black snow pants” and “wearing a black coat” regions.
- Text-to-Image Matching: In Figure 4, the first three columns show BFAN’s top-three retrieved images and the last three show baseline results, with correct answers outlined in green.The figure is organized by text query, allowing direct comparison of retrieval rankings.
5 CONCLUSION
The paper concludes that bidirectional focal attention improves image-text matching by focusing on relevant fragments and reducing semantic misalignment. It reports significant gains over state-of-the-art and identifies other cross-modal applications as future directions.
- Conclusion: BFAN attends only to fragments relevant to the query, addressing semantic misalignment caused by conventional attention.Its directional design also avoids preference for long text or complex images.
- Conclusion: Comprehensive experiments demonstrate that the proposed method significantly outperforms state-of-the-art image-text matching approaches.
- Conclusion: Future work includes applying focal attention to translation, image captioning, and visual question answering.