PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoders

Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, Stéphane Marchand-Maillet

arXiv:2008.05231v2cs.CV

TL;DR

Precise cross-modal matching must preserve local correspondences while remaining suitable for scalable retrieval. TERAN enforces unsupervised word-region alignment and keeps visual and textual pipelines separate until final alignment, achieving state-of-the-art retrieval results on MS-COCO and Flickr30K. The authors conclude that its resulting features support effective scalable retrieval setups.

  • Problem

    Cross-modal retrieval requires fine-grained image-sentence matching, but entangled cross-attention models cannot separately extract visual and textual features for scalable search.

  • Method

    TERAN aligns image regions and sentence words using only global image-sentence supervision, while merging modalities only during the final alignment phase.

  • Results

    TERAN achieves state-of-the-art image and sentence retrieval results on MS-COCO and Flickr30K.

  • Takeaways & Limitations

    TERAN produces effective independent visual and textual features for scalable cross-modal retrieval setups.

  • Takeaways & Limitations

    TERN’s global-scale training can yield low sentence-retrieval performance and random word groundings because it is not forced to learn meaningful fine-grained alignments.

Abstract

from arXiv · show

Despite the evolution of deep-learning-based visual-textual processing systems, precise multi-modal matching remains a challenging task. In this work, we tackle the task of cross-modal retrieval through image-sentence matching based on word-region alignments, using supervision only at the global image-sentence level. Specifically, we present a novel approach called Transformer Encoder Reasoning and Alignment Network (TERAN). TERAN enforces a fine-grained match between the underlying components of images and sentences, i.e., image regions and words, respectively, in order to preserve the informative richness of both modalities. TERAN obtains state-of-the-art results on the image retrieval task on both MS-COCO and Flickr30k datasets. Moreover, on MS-COCO, it also outperforms current approaches on the sentence retrieval task. Focusing on scalable cross-modal information retrieval, TERAN is designed to keep the visual and textual data pipelines well separated. Cross-attention links invalidate any chance to separately extract visual and textual features needed for the online search and the offline indexing steps in large-scale retrieval systems. In this respect, TERAN merges the information from the two domains only during the final alignment phase, immediately before the loss computation. We argue that the fine-grained alignments produced by TERAN pave the way towards the research for effective and efficient methods for large-scale cross-modal information retrieval. We compare the effectiveness of our approach against relevant state-of-the-art methods. On the MS-COCO 1K test set, we obtain an improvement of 5.7% and 3.5% respectively on the image and the sentence retrieval tasks on the Recall@1 metric. The code used for the experiments is publicly available on GitHub at https://github.com/mesnico/TERAN.

1 INTRODUCTION

TERAN addresses difficult image-sentence matching by aligning image regions with sentence words while keeping visual and textual pipelines separate. It reports state-of-the-art retrieval results without region-word supervision.

  • Image-sentence matching requires high-level understanding to score whether a sentence describes an image and retrieve relevant cross-modal items.
  • Existing classification-derived image features can capture only global summaries, overlooking important local information needed for matching.
  • Cross-attention models achieve strong matching results but cannot separately extract visual and textual descriptions for large-scale retrieval.
  • TERAN forces fine-grained word-region alignment using only coarse image-sentence supervision, without explicit correspondence labels.
  • TERAN keeps visual and textual pipelines separate and fuses them only during the final loss computation, enabling scalable retrieval.
  • TERAN produces state-of-the-art image and sentence retrieval results while using independent visual and textual features.

2 RELATED WORK

Related work spans common-space matching, region-word alignment, transformer language models, and disentangled feature pipelines. Strong joint models often sacrifice the separate feature extraction required for scalable search.

  • Image-text matching commonly projects images and sentences into a shared representation space before computing cross-domain similarity.
  • CNNs and recurrent networks have traditionally represented images and sentences, but sentence-level or global image features can discard local information.
  • Transformer-based language models provide contextualized word embeddings and have supported state-of-the-art retrieval and word-region alignment results.
  • Recent joint-processing architectures achieve strong retrieval and alignment performance but cannot produce separate image and caption descriptions for offline indexing and online querying.
  • TERAN differs from related common-space methods by enforcing fine-grained alignment instead of pooling final representations into a single vector.

Cross-Modal Retrieval Evaluation Metrics

Cross-modal retrieval is commonly evaluated with exact-match Recall@K, but real search often requires ranking partially relevant results. The paper therefore extends caption-based relevance evaluation beyond exact matches.

  • Image-caption matching evaluates both retrieving images from captions and retrieving captions from images.
  • The extended evaluation protocol uses caption-based similarity to assign relevance and handle non-exact yet relevant matches.

3 REVIEW OF TRANSFORMER ENCODERS

Transformer encoders use self-attention to transform sets of vectors while modeling relationships among their elements. Their architecture can therefore support both textual sequences and unordered visual-region features.

  • Self-attention weights each input vector using normalized scores computed from the input vectors themselves.
  • The scaling factor √d_k mitigates vanishing gradients when query-key inner products become large.
  • Multi-head attention processes chunks independently, helping capture relationships among different portions of input vectors.
  • A transformer encoder applies feed-forward processing with ReLU, residual connections, and layer normalization while preserving input dimensionality.
  • Because transformer encoders have no built-in sequential positional prior, they can process sets of visual features from an object detector.
  • Stacking encoder instances provides multiple attention levels for deeper reasoning over visual or textual entities.

4 TRANSFORMER ENCODER REASONING AND ALIGNMENT NETWORK (TERAN)

TERAN extends TERN with fine-grained region-word alignment while keeping visual and textual reasoning pipelines separate. It computes image-sentence similarity by pooling affinities between contextualized regions and words, trained with hard-negative ranking.

  • Architecture: TERAN modifies TERN’s objective to enforce fine-grained alignment between image regions and sentence words.The alignment is formed from features output by the final transformer encoder layers.
  • Architecture: Transformer Encoder layers independently process salient image regions and caption words as visual and textual entities.The inputs are sets of regions and sequences of words, respectively.
  • Architecture: TERAN omits the global I-CLS and T-CLS representations used by TERN, instead retaining region- and word-level features.These special tokens remain only for a targeted combined-loss experiment and are excluded from the main TERAN experiments.
  • Alignment: The model constructs a region-word alignment matrix using cosine similarities between contextualized region and word vectors.The matrix indexes regions from an image against words from a sentence.
  • Alignment: Global image-sentence similarity is obtained by pooling the alignment matrix with max-over-regions sum-over-words or its dual max-over-words sum-over-regions.Because these functions are asymmetric, TERAN also evaluates their symmetric sum.
  • Learning: Training uses a hinge-based triplet ranking loss with hard negatives sampled from the mini-batch.The margin specifies the minimum separation between matching and negative pairs.

5 COMPUTATIONAL EFFICIENCY OF TERAN

TERAN is designed for scalable retrieval by independently extracting visual and textual features and using a simple final similarity function. Its feature-space and extraction-time complexity scales additively with database size rather than with image-sentence pairs.

  • Scalable retrieval: TERAN independently forwards visual and textual pipelines and uses a simple similarity function for efficient cross-modal retrieval.This design supports separate feature extraction for indexing and querying.
  • Complexity: O(K) + O(L) feature-space and feature-extraction-time complexity is achieved when K images and L sentences are stored.TERAN shares this additive scaling property with TERN and other separated-pipeline methods.
  • Comparison: O(KL) feature-space and network-evaluation scaling makes entangled visual-textual methods impractical for real-world scalable search engines.Partially separated approaches may still require O(K) or O(L) re-evaluations for a new query.
  • Similarity computation: TERAN’s similarity function uses dot products and summations without complex memory or attention layers.The authors state that this may enable metric-space pruning for efficient rankings.

6 NDCG METRIC FOR CROSS-MODAL RETRIEVAL

The paper supplements Recall@K with NDCG to evaluate ranking quality for cross-modal retrieval. Relevance is derived from sentence similarities, using ROUGE-L and SPICE to support image and caption retrieval evaluation.

  • Motivation: Recall@K measures whether the correct item appears among the first K results, but it is less suited to rankings containing multiple relevant results.The paper motivates a metric that accounts for graded relevance and result positions.
  • NDCG: NDCG evaluates the quality of the first p ranked results while penalizing highly relevant items that appear lower in the list.Normalization by the ideal ranking places NDCG_p in the range [0, 1].
  • NDCG: NDCG assigns each retrieved item a relevance value based on its affinity with the query and normalizes discounted cumulative gain by the ideal ranking.The formula uses relevance values rel_i and the ideal discounted cumulative gain IDCG_p.
  • Relevance construction: For image retrieval, relevance compares the query caption with captions associated with each candidate image; caption retrieval reverses this arrangement.These relevance values are computed from sentence similarity scores.
  • Similarity functions: ROUGE-L and SPICE provide the sentence similarity function, capturing longest common subsequences and syntactic-graph-based semantic similarities, respectively.SPICE is described as having some robustness to synonyms and sensitivity to higher-level textual features.

7 EXPERIMENTS

Experiments evaluate TERAN on MS-COCO and Flickr30k using Recall@K and NDCG, comparing retrieval performance, generalization, alignment variants, and qualitative behavior. TERAN achieves state-of-the-art results while preserving efficient retrieval pipelines and improving fine-grained matching.

  • MS-COCO results: TERAN reaches state-of-the-art results on MS-COCO 1K and 5K test sets across almost all reported metrics.On the 1K test set, the ensemble improves Recall@1 by 5.7% for image retrieval and 3.5% for sentence retrieval over CAMERA; on the 5K set, improvements are 11.3% and 7.6%.
  • Generalization: TERAN generalizes better than TERN on both NDCG and Recall@1 during validation.TERN overfits NDCG while Recall@1 continues improving, whereas TERAN generalizes on both metrics.
  • Alignment analysis: Adding TERAN’s alignment loss slightly improves TERN, indicating that precise region-word alignment benefits fixed-sized global embeddings.The comparison uses region and word outputs for alignment while TERN otherwise learns global image-sentence representations.
  • Flickr30k results: On Flickr30k, TERAN outperforms the best baseline for image retrieval and achieves state-of-the-art ensemble results across all metrics.The ensemble gains 4.6% on image-retrieval Recall@1 and 1.5% on sentence-retrieval Recall@1.
  • Efficiency: TERAN supports efficient reranking, scoring one image-to-sentence query in ∼0.12s and ranking 1K images in ∼0.02s on MS-COCO.These timings were measured on a single GTX 1080Ti and support reranking after faster initial retrieval.
  • Qualitative analysis: Qualitative results show that NDCG better reflects relevant images for ambiguous captions, while exact matching works better for selective captions.For subtle visual details, the correct image can still rank sixth despite visually related higher-ranked results.

8 ABLATION STUDY

The ablation study examines weight sharing, pooling choices, language encoders, and learned word-region alignments. Results favor TERAN’s sum-based pooling and BERT, while visualizations show generally plausible but imperfect groundings.

  • 8.1 The Effect of Weight Sharing: Weight sharing produces comparable TERAN results and a small gain in both Recall@K and NDCG for max-over-words sum-over-regions pooling.The authors interpret the comparable values as evidence that high-level vectors are modality-independent and fully comparable in one representation space.
  • 8.2 Averaging Versus Summing: Averaging instead of summing loses important information and causes premature NDCG overfitting in image retrieval.The fixed number of 36 visual concepts makes average-over-regions and sum-over-regions similar, but averaging over words removes dependence on textual concept count.
  • 8.4 Using Different Language Models: BERT is essential for top Recall@K results: Bi-LSTM and Bi-GRU lose around 14% on image retrieval and 12% on sentence retrieval at Recall@1.The recurrent alternatives retain comparable NDCG performance, especially for image retrieval.
  • 8.5 Visualizing the Visual-Word Alignments: TERAN usually grounds words plausibly, clustering noun phrases and their modifiers on shared image regions despite difficult function words.The alignments are computed by associating each word with its top-relevant region using cosine similarity, without region-word supervision.
  • 8.5 Visualizing the Visual-Word Alignments: Some localized grounding errors remain, such as associating “eyes closed” with a region depicting a closed mouth.The authors nevertheless report that complex scenes are generally decomposed into salient elements and only key regions are attended.

9 CONCLUSIONS

TERAN is a relationship-aware Transformer Encoder architecture that reasons separately over image and text elements while enforcing unsupervised fine-grained region-word alignment. It achieves state-of-the-art retrieval results on MS-COCO and Flickr30K and supports scalable retrieval through effective visual and textual features.

  • 9 CONCLUSIONS: TERAN uses Transformer Encoder self-attention to reason separately about spatial and abstract relationships among image and text elements.The architecture is relationship-aware and processes the two modalities separately.
  • 9 CONCLUSIONS: TERAN forces fine-grained region-word alignment without supervision at that level, unlike TERN’s global matching objective.The alignment objective operates on region and word features rather than only a potentially lossy global representation.
  • 9 CONCLUSIONS: TERAN achieves state-of-the-art results on MS-COCO and Flickr30K while producing effective visual and textual features for scalable retrieval setups.The evaluation uses Recall@K and NDCG with ROUGE-L and SPICE textual relevance measures.
  • 9 CONCLUSIONS: The paper presents TERAN as a path toward efficient and effective cross-modal information retrieval.This conclusion follows the combination of fine-grained alignment and separable visual-textual feature extraction.
Loading 2008.05231v2…