PaperlayerView exact revision on arXiv

Source-linked AI summary

Similarity Reasoning and Filtration for Image-Text Matching

Haiwen Diao, Ying Zhang, Lin Ma, Huchuan Lu

arXiv:2101.01368v1cs.CVcs.MM

TL;DR

Image-text matching remains challenging because complex matching patterns and visual-textual semantic discrepancies make alignment use underexplored. SGRAF learns vector-based local and global similarity representations, applies graph reasoning to their relationships, and filters alignments by significance. Experiments report state-of-the-art performance on Flickr30K and MSCOCO, alongside qualitative interpretability for SGR and SAF.

  • Problem

    Image-text matching must infer accurate visual-semantic similarity despite complex matching patterns, large semantic discrepancies, and limitations in representing and aggregating local and global alignments.

  • Method

    SGRAF learns vector-based similarity representations, uses SGR to reason over local-global alignment relationships, and uses SAF to attend selectively to meaningful alignments.

  • Results

    The method achieves state-of-the-art performance on Flickr30K and MSCOCO and shows good interpretability of SGR and SAF through qualitative experiments and analyses.

  • Takeaways & Limitations

    Effective image-text matching benefits from exploiting relationships among local and global alignments while suppressing disturbances from less-meaningful alignments.

Abstract

from arXiv · show

Image-text matching plays a critical role in bridging the vision and language, and great progress has been made by exploiting the global alignment between image and sentence, or local alignments between regions and words. However, how to make the most of these alignments to infer more accurate matching scores is still underexplored. In this paper, we propose a novel Similarity Graph Reasoning and Attention Filtration (SGRAF) network for image-text matching. Specifically, the vector-based similarity representations are firstly learned to characterize the local and global alignments in a more comprehensive manner, and then the Similarity Graph Reasoning (SGR) module relying on one graph convolutional neural network is introduced to infer relation-aware similarities with both the local and global alignments. The Similarity Attention Filtration (SAF) module is further developed to integrate these alignments effectively by selectively attending on the significant and representative alignments and meanwhile casting aside the interferences of non-meaningful alignments. We demonstrate the superiority of the proposed method with achieving state-of-the-art performances on the Flickr30K and MSCOCO datasets, and the good interpretability of SGR and SAF modules with extensive qualitative experiments and analyses.

Introduction

Image-text matching remains difficult because visual-language pairs exhibit complex matching patterns and large semantic discrepancies. SGRAF addresses limitations in similarity representation, alignment reasoning, and alignment aggregation.

  • Image-text matching measures visual-semantic similarity and supports tasks including cross-modal retrieval, image captioning, text-to-image synthesis, and multimodal translation.
  • Previous methods often miss local interactions between image regions and sentence fragments, limiting interpretability and performance gains.
  • Scalar cosine similarities may inadequately characterize region-word association patterns.
  • Max or average pooling can hinder information communication between local and global alignments and retain distractions from less-meaningful alignments.Examples include alignments involving words such as “a” and “in”.
  • SGRAF learns vector-based similarity representations and combines SGR graph reasoning with SAF attention filtration for image-text matching.SGR captures relationships between local and global alignments, while SAF assigns significance-aware attention to reduce irrelevant interactions.

Related Work

Related work addresses image-text matching through feature encoding, similarity prediction, graph reasoning, and attention mechanisms. SGRAF differs by targeting cross-modal similarity reasoning over local and global alignments.

  • Feature Encoding: Prior feature-encoding methods extract textual and visual representations using techniques including word embeddings, recurrent networks, self-attention, and region-based visual features.
  • Similarity Prediction: Most image-text matching methods learn joint embeddings and similarity measures, often using inner products or cosine distance for global alignments.
  • Graph Reasoning: Graph neural networks have modeled dependencies between concepts in visual-semantic tasks such as captioning, VQA, and referring-expression grounding.
  • Graph Reasoning: Existing scene-graph approaches refine single-modality contextualized representations, whereas SGR explores complex matching patterns through global and local cross-modal alignments.
  • Attention Mechanism: Attention methods adaptively attend to visual or textual parts and have been applied to latent region-word alignments in image-text matching.

Method

The method encodes images and sentences into local and global features, computes vector-based cross-modal similarity representations, and uses SGR and SAF to reason over and aggregate alignments.

  • Overall Architecture: The method begins by encoding visual and textual observations, then computes similarity representations for local and global representation pairs.
  • Feature Encoding: Images provide K region-level features transformed into d-dimensional local vectors, while self-attention aggregates regions into a global visual representation.
  • Feature Encoding: Sentences are tokenized into L words, encoded with a bidirectional GRU, and pooled with self-attention into local and global textual representations.
  • Vector Similarity Function: Vector-based similarity representations replace scalar similarity measures to capture more detailed associations between feature representations from different modalities.A learnable matrix W produces an m-dimensional similarity vector.
  • Local Similarity Representation: Textual-to-visual attention creates word-specific attended visual features before computing fine-grained local similarity representations.Attention weights use a temperature parameter and region-word cosine similarities.
  • Similarity Reasoning and Filtration: SGR reasons over relationships among local and global alignments, while SAF attends to informative alignments for more accurate similarity prediction.

Similarity Graph Reasoning

SGR represents local and global alignments as nodes in a similarity graph, propagates information through directed edges, and uses the final global node to predict image-text similarity.

  • SGR constructs graph nodes from word-attended local similarity representations and one global similarity representation.
  • The graph uses directed edges with separate incoming and outgoing linear transformations to support complex similarity information propagation.
  • At each reasoning step, connected node features are aggregated and nonlinearly transformed to update the current node representation.
  • After N reasoning steps, SGR feeds the final global node representation into a fully connected layer to infer the final similarity score.
  • Information propagation between local and global alignments enables more comprehensive interactions for similarity prediction.

Similarity Attention Filtration

SAF assigns learned aggregation weights to local and global similarity representations, enhancing informative alignments while suppressing irrelevant interactions before final similarity prediction.

  • Less-meaningful local alignments can hinder similarity aggregation when all possible alignments receive undifferentiated treatment.
  • SAF computes an aggregation weight for each local or global similarity representation using a sigmoid-based transformation with batch normalization.
  • The module aggregates similarity representations with their learned weights and feeds the result into a fully connected layer for final similarity prediction.
  • SAF increases the contribution of informative representations while reducing disturbance from less-meaningful alignments such as those involving “the” or “be”.
  • SGR and SAF can be trained jointly or independently, with their predicted similarities averaged during retrieval inference.

Experiments

The experiments evaluate SGRAF on Flickr30K and MSCOCO using bidirectional retrieval metrics and configuration studies. Results show strong retrieval performance and benefits from graph reasoning, vector-based similarities, attention filtration, and independent training.

  • Evaluation Setup: SGRAF is evaluated on MSCOCO and Flickr30K using Recall at K, including R@1, R@5, and R@10.MSCOCO results are reported for 1K and 5K test-image settings.
  • Retrieval Results: 79.6% R@1 for sentence retrieval and 63.2% R@1 for image retrieval are achieved on MSCOCO with 1K test images.On 5K test images, the approach maintains superiority with more than 3% improvement on R@1.
  • SGR Configurations: 12.8% improvement for sentence retrieval and 10.2% for image retrieval is obtained by SGR over the graph-reasoning baseline on R@1.The study also sets graph reasoning to three steps for maximum performance and finds global similarity improves local-similarity aggregation.
  • Similarity Configurations: Vector-based similarity representations outperform local cosine-distance averaging, while T2I attention is more effective for SGR than I2T attention.The authors attribute weaker I2T SGR gains to redundant, relatively independent, and irregularly ordered region features.
  • Similarity Configurations: SAF improves both I2T and T2I configurations by filtering and aggregating discriminative local alignments.This supports using attention filtration to improve image-text matching precision.
  • Training Configurations: Independent training gives SGRAF an exact and impressive promotion, while joint training selectively benefits SAF sentence retrieval and SGR image retrieval.The authors suggest the modules capture complementary alignment information but remain susceptible to different difficulties.
  • Qualitative Analysis: SAF highlights meaningful cues and suppresses irrelevant interactions, while SGR reinforces discrepant alignments in hard negatives.In the example, SAF exposes the unmatched word “brown,” and SGR lowers the hard-negative similarity.

Conclusion

The paper presents SGRAF as a combination of similarity graph reasoning and attention filtration for image-text matching. Experiments support exploiting local-global alignment relationships and suppressing less-meaningful alignments, with further improvements when SGR and SAF cooperate.

  • Conclusion: SGR performs multistep reasoning over global and local similarity nodes, while SAF attends to discriminative alignments for similarity aggregation.Both modules are components of the proposed SGRAF network.
  • Conclusion: Extensive benchmark experiments show that SGR and SAF discover image-text associations and improve further when cooperating.The conclusion specifically emphasizes local-global relationships and suppression of less-meaningful alignments.

Appendix Overview

The supplementary document extends the paper with additional diagrams, quantitative studies, and qualitative retrieval analyses of SGRAF.

  • Supplementary Contents: The appendix adds diagrams and descriptions of the SGRAF network, including self-attention and the SGR module.
  • Supplementary Contents: It reports quantitative studies on graph dimension and qualitative retrieval examples for bidirectional retrieval and model visualization.

Network Details

The network details describe generic representation extraction, similarity representation learning, graph reasoning, and attention filtration. SGR updates similarity nodes through learned graph propagation, while SAF weights and aggregates them into a scalar similarity.

  • Implementation Components: SGRAF implementation details cover generic representation extraction, similarity representation learning, similarity graph reasoning, and attention filtration.
  • Generic Representation Extraction: The model extracts 36 region proposals with 2048-dimensional features, transforms them to 1024-dimensional vectors, and uses self-attention for a global visual vector.Words are represented with 300-dimensional embeddings and encoded with a Bi-GRU.
  • Similarity Graph Reasoning: SGR treats L+1 256-dimensional similarity vectors as graph nodes and updates each node by learned neighbor aggregation followed by nonlinear transformation.Each node aggregates connected nodes, including itself, across reasoning steps.
  • Similarity Attention Filtration: SAF learns L+1 attention weights over the similarity vectors, aggregates them into one 256-dimensional vector, and predicts scalar similarity through a fully connected layer and sigmoid.

Quantitative Studies

The quantitative study examines how graph dimension affects SGR on Flickr30K, using three reasoning steps and separate parameters at each step. The results indicate robustness across similarity-representation dimensions, with m=256 selected for best image-text retrieval performance.

  • Graph Dimension: SGR is insensitive to the similarity-representation dimension, suggesting stabilization and robustness across tested graph dimensions.The evaluation uses three reasoning steps, with parameters not shared across steps.
  • Graph Dimension: m=256 is selected as the graph dimension because it yields the best image-text retrieval results.
  • Evaluation Setup: The graph-dimension evaluation reports results for both global and local alignments on Flickr30K.Table 7 presents the impact of graph dimension on Flickr30K.
  • Notation: L denotes the number of words in a sentence and the number of local alignments attended by textual words.

Qualitative Studies

The qualitative studies show SGRAF retrieving relevant sentences and images, including difficult cases with closely related semantics. Visualizations indicate that SAF emphasizes discriminative alignments while SGR highlights fine-grained cues and separates matched from unmatched pairs.

  • Retrieval Examples: SGRAF efficiently retrieves correct sentences and accurately identifies ground-truth images despite hard negative samples with similar semantics.The qualitative retrieval examples cover both sentence retrieval and image retrieval on Flickr30K and MSCOCO.
  • SGRAF Visualization: SAF highlights discriminative alignments such as “two man” and “dancing” while suppressing function-word alignments such as “the” and “in.”
  • SGR Visualization: The SGR visualization represents local and global similarities as graph nodes and iteratively reasons over them for three steps.Edges are computed from learned representations and normalized with a row-wise softmax before node aggregation.
  • SGRAF Visualization: SGR stresses cues such as “young boy” and “Texas,” producing larger gaps between matched and unmatched pairs.
  • Retrieval Examples: Top-5 retrieved results are displayed for sentence retrieval, with green marking ground-truth sentences and red marking unmatched retrievals.
  • Retrieval Examples: Top-5 retrieved results are displayed for image retrieval, with green marking ground-truth images and red marking unmatched retrievals.
Loading 2101.01368v1…