Source-linked AI summary
Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
Haoran Wang, Ying Zhang, Zhong Ji, Yanwei Pang, Lin Ma
TL;DR
Image-text matching is limited by reliance on instance-pair associations and by the semantic discrepancy between image and text. CVSE incorporates shared commonsense knowledge through consensus-aware concept representations learned from concept co-occurrences, then combines consensus-level and instance-level representations for alignment. Experiments on MSCOCO and Flickr30k show that CVSE outperforms state-of-the-art models for image-text retrieval.
Problem
Existing image-text matching methods mainly exploit instance-pair associations without external commonsense knowledge, while images and texts occupy heterogeneous representation spaces.
Method
CVSE builds a concept correlation graph from semantic-concept co-occurrences in an image-caption corpus, learns consensus-aware concept representations, and integrates them with instance-level representations.
Results
CVSE outperforms state-of-the-art models for image-text retrieval on the MSCOCO and Flickr30k datasets.
Takeaways & Limitations
Consensus knowledge provides unified high-level representations for both modalities and supports more meaningful visual-semantic embeddings for retrieval.
Takeaways & Limitations
The co-occurrence-derived concept correlation matrix may deviate from real-world data distributions and be biased by long-tail concept frequencies.
Abstract
from arXiv · showhide
Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations, thereby exploiting their matching relationships and making the corresponding alignments. Such approaches only exploit the superficial associations contained in the instance pairwise data, with no consideration of any external commonsense knowledge, which may hinder their capabilities to reason the higher-level relationships between image and text. In this paper, we propose a Consensus-aware Visual-Semantic Embedding (CVSE) model to incorporate the consensus information, namely the commonsense knowledge shared between both modalities, into image-text matching. Specifically, the consensus information is exploited by computing the statistical co-occurrence correlations between the semantic concepts from the image captioning corpus and deploying the constructed concept correlation graph to yield the consensus-aware concept (CAC) representations. Afterwards, CVSE learns the associations and alignments between image and text based on the exploited consensus as well as the instance-level representations for both modalities. Extensive experiments conducted on two public datasets verify that the exploited consensus makes significant contributions to constructing more meaningful visual-semantic embeddings, with the superior performances over the state-of-the-art approaches on the bidirectional image and text retrieval task. Our code of this paper is available at: https://github.com/BruceW91/CVSE.
1 Introduction
Image-text matching must bridge heterogeneous image and text spaces, but instance-level methods often miss shared commonsense relationships. CVSE incorporates consensus-aware concept representations alongside instance-level representations to improve cross-modal alignment and retrieval.
- Image-text matching retrieves texts or images describing the most relevant contents for a query, despite semantic discrepancies between heterogeneous representation spaces.
- Global-representation methods overlook relationships between image objects and sentence segments, while fragment-level methods perform more complicated local matching.
- Consensus captures commonsense associations among semantic concepts, such as the likely co-occurrence of “surfboard” and “wave” in related images and captions.
- CVSE computes concept co-occurrence correlations from an image-caption corpus, constructs a concept correlation graph, and learns consensus-aware concept representations with graph convolution.
- CVSE integrates consensus-level and instance-level representations to align images and texts for bidirectional retrieval.
- CVSE outperforms state-of-the-art methods on traditional image-text retrieval and shows superior generalization for cross-domain transfer across two benchmark datasets.
2 Related Work
Prior vision-language methods use external knowledge in selected tasks or align image-text representations globally or locally. CVSE instead uses consensus-aware representations to uniformly align both modalities through shared high-level semantics.
- External knowledge has been applied to image classification, object recognition, visual question answering, and scene graph generation.
- Image-text matching studies commonly use two-branch architectures with global or local representations aligned in a joint semantic space.
- Unlike scene-graph-based visual knowledge methods, CVSE uses learned consensus-aware concept representations to uniformly represent and align images and texts.
- CVSE measures cross-modal similarity by disentangling higher-level semantics for both modalities, improving interpretability.
3 Consensus-Aware Visual-Semantic Embedding
CVSE augments instance-level image-text matching with consensus-level representations learned from commonsense concept relationships. It builds consensus-aware concept representations from caption-corpus co-occurrences, then combines them with modality-specific instance representations for alignment.
- Consensus-Aware Concept Representation Learning: CVSE introduces a Consensus Exploitation module that captures semantic associations among concepts using graph convolution over a correlation graph.The module propagates information through concept neighborhoods to inject commonsense knowledge into concept representation learning.
- Consensus-Aware Concept Representation Learning: The concept vocabulary is selected from frequent words in an image-captioning corpus and instantiated with GloVe embeddings.Selected concepts are roughly categorized as objects, motions, and properties with a 7:2:1 ratio.
- Consensus-Aware Concept Representation Learning: CVSE models concept relationships with an asymmetric conditional-probability matrix derived from concept co-occurrences.Asymmetry captures directional interdependencies rather than simple co-occurrence frequency.
- Consensus-Aware Concept Representation Learning: Confidence scaling and thresholded binarization focus the graph on reliable concept relationships while reducing long-tail noise and overfitting.The scaling function adjusts correlation strengths, and threshold ε filters noisy edges.
- Consensus-Aware Concept Representation Learning: Stacked graph convolutional layers produce consensus-aware concept representations that encode higher-order neighborhood information.The resulting CAC matrix provides embeddings for the concept vocabulary in the joint embedding space.
- Consensus-Aware Representation Learning: CVSE queries CAC representations with instance-level image and text representations, then fuses consensus-level and instance-level representations for both modalities.Concept scores guide linear combinations of CAC embeddings, while concept labels provide prior information for text consensus representation learning.
4 Experiments
Experiments on MSCOCO and Flickr30k evaluate CVSE using recall-based retrieval metrics and ablations of consensus exploitation, training objectives, inference, and representation levels. Results show strong retrieval performance, benefits from consensus components and combined semantic levels, and useful cross-domain and concept-level behavior.
- Comparison to State-of-the-art: 74.8% R@1 for text retrieval and 59.9% R@1 for image retrieval are achieved on MSCOCO, with CVSE exceeding competitors on most metrics.Compared with LIWE, image-retrieval gains are 2.0%, 1.1%, and 1.0% on R@1, R@5, and R@10.
- Comparison to State-of-the-art: 80.4% mR is achieved on Flickr30k, while text-retrieval improvements over the previous best are 3.9%, 1.8%, and 0.2% on R@1, R@5, and R@10.The results are reported on the Flickr30k test set.
- Ablation Studies: Removing the consensus-similarity component reduces R@1 by 0.3% for text retrieval and 1.1% for image retrieval.Using only GloVe-based CAC representations remains comparable to leading methods, while excluding concept labels causes an evident performance drop.
- Ablation Studies: Removing the separate ranking losses or KL-divergence regularization produces inferior retrieval results.The KL term regularizes discrepancies between image and text predicted concept-score distributions.
- Ablation Studies: Combining instance-level and consensus-level representations outperforms using either semantic level alone.The ablation supports complementarity between the two representation levels.
- Ablation Studies: Cross-dataset experiments transfer a model trained on MSCOCO directly to Flickr30k to evaluate consensus knowledge for domain adaptation.The paper frames this setting as a test of cross-modal retrieval generalization in a real scenario.
- Further Analysis: Figure 4 shows predicted concept scores capturing informative concepts absent from the image-text pair, such as “traffic” and “buildings.”These associated concepts are used to enhance semantic representations.
- Further Analysis: Figure 5 visualizes word2vec-based and consensus-aware concept representations with t-SNE, using 15 randomly selected concepts per part of speech and shared colors by part of speech.The visualization focuses on comparing the two concept-representation spaces.
5 Conclusions
CVSE integrates commonsense consensus knowledge into multimodal representation learning to improve image-text association. It outperforms state-of-the-art models on MSCOCO and Flickr30k image-text retrieval.
- CVSE integrates commonsense knowledge into multimodal representation learning for visual-semantic embedding.
- The model exploits consensus knowledge to identify high-level concepts and generate unified consensus-aware concept representations for images and text.
- CVSE outperforms state-of-the-art models on image-text retrieval using the MSCOCO and Flickr30k datasets.