Source-linked AI summary
Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval
Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, Xilin Chen
TL;DR
Image-text retrieval in complex scenes needs representations that capture both objects and their relationships across heterogeneous modalities. The paper uses visual and textual scene graphs, tailored graph encoders, and two-level matching, reporting state-of-the-art results on Flickr30k and MSCOCO. The authors also identify a failure mode in which the model can overemphasize relationships.
Problem
Complex image-text retrieval requires matching multiple objects and their relationships across heterogeneous modalities, while existing approaches may omit explicit relationship modeling.
Method
The Scene Graph Matching model builds visual and textual scene graphs, encodes them with tailored graph encoders, and matches their object-level and relationship-level features.
Results
The method outperforms state-of-the-art image-text retrieval methods on both Flickr30k and MSCOCO.
Takeaways & Limitations
Explicitly modeling relationships supports more comprehensive image-text matching, and qualitative results show that retrieved images change when query relationship words change.
Takeaways & Limitations
A failure case shows that SGM can focus too much on relationships, leaving the balance between object and relationship emphasis for future work.
Abstract
from arXiv · showhide
Image-text retrieval of natural scenes has been a popular research topic. Since image and text are heterogeneous cross-modal data, one of the key challenges is how to learn comprehensive yet unified representations to express the multi-modal data. A natural scene image mainly involves two kinds of visual concepts, objects and their relationships, which are equally essential to image-text retrieval. Therefore, a good representation should account for both of them. In the light of recent success of scene graph in many CV and NLP tasks for describing complex natural scenes, we propose to represent image and text with two kinds of scene graphs: visual scene graph (VSG) and textual scene graph (TSG), each of which is exploited to jointly characterize objects and relationships in the corresponding modality. The image-text retrieval task is then naturally formulated as cross-modal scene graph matching. Specifically, we design two particular scene graph encoders in our model for VSG and TSG, which can refine the representation of each node on the graph by aggregating neighborhood information. As a result, both object-level and relationship-level cross-modal features can be obtained, which favorably enables us to evaluate the similarity of image and text in the two levels in a more plausible way. We achieve state-of-the-art results on Flickr30k and MSCOCO, which verifies the advantages of our graph matching based approach for image-text retrieval.
1. Introduction
Image-text retrieval must match heterogeneous modalities while preserving multiple objects and their semantic relationships. The paper represents images and text with scene graphs and evaluates object- and relationship-level similarity.
- Global representations overlook local details, making retrieval less satisfactory for complex natural scenes with multiple objects.
- Objects alone may be insufficient because relationships distinguish scenes containing similar objects, such as standing next to versus riding on.
- The method introduces visual scene graphs for images and textual scene graphs for text, converting retrieval into cross-modal scene graph matching.
- Two tailored graph encoders refine node representations and produce object-level and relationship-level features for matching.
- Experiments on Flickr30k and MSCOCO report performance that significantly outperforms state-of-the-art methods.
2. Related Works
Prior image-text retrieval methods use global or local representations, but several do not explicitly model relationships between objects. This paper addresses that gap with scene graphs for two-level cross-modal matching.
- Image-text retrieval methods are grouped into global-representation and local-representation approaches.
- Global methods encode whole images and sentences into features and compare their similarity.
- Local methods align image objects or regions with sentence words or fragments, sometimes using attention to emphasize important elements.
- Some existing methods use contextual word embeddings without explicitly revealing semantic relationships between textual objects.
- The proposed approach explicitly captures visual and textual objects with their relationships, enabling matching at two levels.
3. Method
The Scene Graph Matching model parses an image and sentence into separate scene graphs and encodes them into feature graphs. It then compares the modalities at object and relationship levels.
- SGM evaluates image-text similarity by dissecting each input into a visual or textual scene graph.
- The visual branch converts a visual scene graph into a visual feature graph, while the textual branch converts a textual scene graph into a textual feature graph.
- The resulting feature graphs are matched at object and relationship levels.
3.1. Visual Feature Embedding
Visual feature embedding represents visual scene-graph nodes with region features and labels, fuses these modalities, and applies graph convolution with distinct updates for object and relationship nodes.
- Visual Scene Graph Generation: A visual scene graph contains object nodes tied to image regions and relationship nodes connecting object pairs by directed edges.
- Visual Scene Graph Generation: Object and relationship nodes carry category labels, represented as one-hot vectors for their respective categories.
- Visual Scene Graph Encoder: The MGCN combines a visual feature extractor, label embedding layer, multimodal fusion layer, and graph convolution network.
- Visual Feature Extractor: Visual features encode image regions, with relationship nodes using the union region of their connected objects.
- Multi-modal Fusion Layer: Visual and label features are concatenated and fused into a unified multimodal feature graph.
- Graph Convolution Network: The GCN aggregates graph information while updating object and relationship nodes in different manners to produce the encoded visual feature graph.
3.2. Textual Feature Embedding
The textual scene graph represents sentence words and semantic relationships with distinct edge types and encodes their paths to obtain object- and relationship-level features.
- Textual Scene Graph Generation: A textual scene graph contains word-order edges and semantic relationship edges parsed from semantic triplets such as “man-hold-baby”.Word-order edges connect words according to sentence order, while semantic edges connect related words beyond grammatical structure.
- Textual Scene Graph Encoder: The textual scene graph encoder uses a word embedding layer, a word-level bi-GRU, and a path-level bi-GRU.The two path types are encoded separately by different bi-GRUs.
- Textual Scene Graph Encoder: The word-level bi-GRU encodes nodes along the word-order path to generate contextual object-level features at hidden states.Each word is first mapped to a vector by the word embedding layer.
- Textual Scene Graph Encoder: The path-level bi-GRU encodes semantic relationship paths to obtain explicit relationship-level features beyond sentence grammar.Each semantic relationship path produces a relationship feature for the textual scene graph.
3.3. Similarity Function
The similarity function separately matches object and relationship features across visual and textual feature graphs, then combines their scores.
- Object- and Relationship-level Matching: Object-level similarity compares every visual and textual object feature to form an Nw × No score matrix.Each feature is a D-dimension vector, and vector similarity is computed using h_i^T h_j.
- Object- and Relationship-level Matching: For each textual object, the maximum similarity among visual objects is selected and averaged as the object-level graph score.This selects the most related visual object for every textual object.
- Object- and Relationship-level Matching: Relationship-level similarity is calculated in the same way as object-level similarity.The two modalities therefore receive corresponding object- and relationship-level scores.
- Combined Similarity: The final visual-textual graph similarity is defined as S = So + Sr.So is the object-level score and Sr is the relationship-level score.
3.4. Loss Function
The loss function uses margin-based triplet learning with hardest negatives from each mini-batch to separate matched from unmatched image-text pairs.
- Triplet Loss: Triplet loss constrains matched image-text pairs to score higher than unmatched pairs by a margin m.The loss compares a corresponding image-sentence pair with non-corresponding pairs.
- Triplet Loss: The corresponding pair consists of image k and sentence l, while image k with sentence l̂ and image k̂ with sentence l are non-corresponding pairs.The hats indicate the unmatched image or sentence in the mini-batch.
- Hardest Negatives: Hardest-negative training selects the highest-scoring non-corresponding image or sentence rather than using all negative samples.The hardest negatives are defined by arg max over non-matching samples.
4. Experiments
Experiments evaluate SGM on Flickr30k and MSCOCO using standard retrieval metrics, ablations, state-of-the-art comparisons, and qualitative analyses. Results support modeling relationships in both modalities for cross-modal retrieval.
- 4.1. Datasets and Evaluation Metrics: Experiments use Flickr30k and MSCOCO for caption retrieval and image retrieval, evaluated with R@k and Medr.R@k measures the percentage of queries whose ground truth appears within the top K; Medr is the median rank of the first retrieved ground truth.
- 4.2. Implementation Details: The visual scene graph generator recognizes 150 object categories and 50 relationship categories, retaining 36 objects and 25 relationships by confidence.
- 4.3. Ablation Study: SGM is compared with object-only and relationship-augmented variants to assess the contribution of visual and textual relationships.OOM matches only objects; OOM w VRel and OOM w TRel add visual and textual relationships separately, while OOM w/o TCxt isolates elements without textual context.
- 4.3. Ablation Study: Adding relationship information in both modalities produces obvious improvements, especially in R@1, for both image retrieval and caption retrieval.
- 4.3. Ablation Study: Relationships improve modality-specific retrieval, while matching object-level and relationship-level features in both modalities yields a large cross-modal retrieval improvement.
- 4.4. Comparison with State-of-the-art Methods: SGM achieves new state-of-the-art results on Flickr30k and MSCOCO, including relative R@1 gains of 16.8% and 16.18% on Flickr30k.On MSCOCO 5k test images, relative R@1 improvements are 10.62% for caption retrieval and 6.65% for image retrieval.
5. Conclusion
The work represents images and text as visual and textual scene graphs, then uses graph encoders to extract object- and relationship-level features for matching. Experiments on Flickr30k and MSCOCO show that explicitly modeling relationships improves image-text retrieval.
- SGM represents images and text as visual and textual scene graphs containing objects and their relationships.
- Two graph encoders extract object-level and relationship-level features from the graphs for image-text matching.
- SGM outperforms state-of-the-art methods in image-text retrieval experiments on both Flickr30k and MSCOCO.
- Qualitative experiments show that SGM captures relationships and is helpful for image-text retrieval.
Supplementary Material: Cross-modal Scene Graph Matching for
Supplementary results compare SGM with object-only matching and examine relationship-sensitive retrieval behavior. They also document a failure case in which SGM focuses too much on relationships.
- A failure case shows that SGM can focus too much on relationships, leaving the balance between objects and relationships as future work.
- When relationship words in queries are modified, SGM's retrieved results change substantially.
- SGM is compared with OOM using qualitative image retrieval results on MSCOCO, with red bounding boxes marking ground truth.
- Figure 2 presents a failure case of SGM and OOM, with the red bounding box identifying the ground-truth image.
- Figure 3 compares the top-5 retrieved results before and after modifying relationship words in queries.