Source-linked AI summary
Graph Structured Network for Image-Text Matching
Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Bin Wang, Yongdong Zhang
TL;DR
Image-text matching needs correspondence that captures more than coarse object co-occurrence, particularly relations and attributes. GSMN models these elements as structured phrases, matches nodes across visual and textual graphs, and propagates associations to infer phrase correspondence. It reports relative Recall@1 improvements of nearly 7% on Flickr30K and 2% on MSCOCO over state-of-the-art methods.
Problem
Existing image-text matching methods learn coarse object-based correspondence while failing to capture fine-grained phrase correspondence involving relations and attributes.
Method
GSMN matches heterogeneous visual and textual graphs using node-level matching followed by structure-level propagation of node correspondence to infer fine-grained phrase correspondence.
Results
The method outperforms state-of-the-art approaches on Flickr30K and MSCOCO, with relative Recall@1 improvements of nearly 7% and 2%, respectively.
Takeaways & Limitations
Modeling objects, relations, and attributes jointly improves correspondence learning and image-text matching performance.
Abstract
from arXiv · showhide
Image-text matching has received growing interest since it bridges vision and language. The key challenge lies in how to learn correspondence between image and text. Existing works learn coarse correspondence based on object co-occurrence statistics, while failing to learn fine-grained phrase correspondence. In this paper, we present a novel Graph Structured Matching Network (GSMN) to learn fine-grained correspondence. The GSMN explicitly models object, relation and attribute as a structured phrase, which not only allows to learn correspondence of object, relation and attribute separately, but also benefits to learn fine-grained correspondence of structured phrase. This is achieved by node-level matching and structure-level matching. The node-level matching associates each node with its relevant nodes from another modality, where the node can be object, relation or attribute. The associated nodes then jointly infer fine-grained correspondence by fusing neighborhood associations at structure-level matching. Comprehensive experiments show that GSMN outperforms state-of-the-art methods on benchmarks, with relative Recall@1 improvements of nearly 7% and 2% on Flickr30K and MSCOCO, respectively. Code will be released at: https://github.com/CrossmodalGroup/GSMN.
1. Introduction
Image-text matching seeks accurate cross-modal correspondence, but existing methods largely miss fine-grained relations and attributes. GSMN addresses this by matching structured visual and textual phrases through graph-based correspondence learning.
- Image-text matching aims to match instances across modalities and accurately reflect image-text pair similarity.
- Existing methods learn coarse object-based correspondence, making relation and attribute matching difficult and allowing objects to map to incorrect categories.
- GSMN explicitly models objects, relations, and attributes as structured phrases and jointly infers their fine-grained correspondence.
- The network constructs graphs for image and text, performs node-level matching, and propagates correspondence to neighboring nodes for structure-level phrase matching.
- The paper presents graph-based matching on heterogeneous visual and textual graphs and reports extensive experiments on Flickr30K and MSCOCO.
2. Related Work
Prior image-text matching methods primarily learn global image-sentence or local region-word correspondence. GSMN extends this framework by representing visual and textual content as graphs whose nodes encode objects, relations, and attributes.
- Existing approaches: Related methods are broadly divided into global correspondence learning and local correspondence learning.
- Global correspondence: Global methods project whole-image and sentence representations into a common space, often optimizing similarity with ranking loss.
- Local correspondence: Local methods learn region-word correspondence using salient objects, selected region-word pairs, semantic concepts, or recurrent selection.
- Graph structured matching: GSMN instead builds graphs with object, relation, and attribute nodes connected when nodes are semantically dependent.
- Local correspondence: Other local approaches discover multiple region-word correspondences and may jointly model global image-text and local region-word mappings.
3. Method
GSMN constructs visual and textual graphs, then matches nodes across modalities and propagates their correspondences through graph neighborhoods to infer fine-grained phrase correspondence. The resulting phrase-level matches are combined into a global image-text similarity and trained with hard-negative triplet loss.
- Graph Construction: GSMN constructs textual graphs from semantic dependencies among words and visual fully-connected graphs from salient image regions.Textual nodes represent words, while visual nodes are salient regions detected by Faster-RCNN.
- Node-level Matching: Node-level matching associates each textual or visual node with nodes from the other modality using similarity-based weighted feature aggregation.For textual nodes, visual features are aggregated with softmax-derived correspondence weights; the operation is symmetric for visual nodes.
- Node-level Matching: A multi-block module converts each node-to-aggregated-node comparison into a matching vector whose blocks can play different roles.The resulting vector is propagated to neighboring nodes to guide fine-grained phrase correspondence.
- Structure-level Matching: Structure-level matching updates node matching vectors by integrating neighborhood vectors with graph convolution, producing convolved vectors that reflect correspondences of connected nodes.Connected nodes form localized phrases, allowing neighboring object, relation, and attribute correspondences to jointly guide phrase matching.
- Global Matching Score: An MLP combines convolved phrase vectors into directional matching scores, and the image-text similarity sums scores from both graph-matching directions.The model performs structure-level matching on both visual and textual graphs.
- Objective Function: Training uses triplet loss with matched pairs as positives and mismatched pairs as negatives, emphasizing hard negatives with the highest loss.Positive similarity is required to exceed negative similarity by margin γ in both text-to-image and image-to-text retrieval.
4. Experiment
Experiments on Flickr30K and MSCOCO evaluate GSMN against established baselines, analyze network structures and λ settings, and visualize learned correspondences. GSMN achieves stronger retrieval performance while modeling fine-grained phrase correspondence.
- Experimental Setup: GSMN is evaluated on Flickr30K and MSCOCO using Recall@1, Recall@5, Recall@10, and rSum for both retrieval directions.The benchmarks contain paired images and five corresponding sentences per image; rSum aggregates Recall values across image-to-text and text-to-image retrieval.
- Experimental Setup: The comparisons include global correspondence, optimization-based, region-word, and state-of-the-art methods, including PFAN, SCAN, and BFAN.The study also compares dense and sparse GSMN variants and reports results against ensemble baselines.
- Benchmark Results: 6.4% and 7% relative R@1 gains over PFAN are reported for image-to-text and text-to-image matching on Flickr30K.GSMN outperforms state-of-the-art methods across all evaluation metrics on this benchmark, and its single model exceeds an ensemble baseline.
- Benchmark Results: Nearly 2% Recall@1 improvement is reported on MSCOCO, while GSMN exceeds SCAN by over 5.5% and 4.5% relative Recall@1 in the two directions.The sparse model performs better than the dense model on MSCOCO, whereas image-to-text Recall@10 is slightly below PFAN because of noise.
- Parameter Analysis: The scaling factor λ controls node weighting and textual edge weighting, with λ = 20 performing best on Flickr30K and λ = 10 on MSCOCO.The authors attribute the dataset-dependent preference to different data distributions; small λ values cannot distinguish relevant from irrelevant nodes effectively.
- Qualitative Analysis: Visualizations show node associations, improved phrase correspondence, high-ranked retrievals, and fine-grained relation and attribute matching.The Flickr30K case study reports that the model can distinguish different kinds of hats in a text-to-image query.