Source-linked AI summary
Learning Semantic Concepts and Order for Image and Sentence Matching
Yan Huang, Qi Wu, Liang Wang
TL;DR
Image and sentence matching remains difficult because pixel-level image representations lack the high-level semantics and ordering expressed in matched sentences. The paper learns regional semantic concepts and organizes them with global context and sentence-order supervision, then jointly trains matching and generation; experiments report state-of-the-art results on two public datasets.
Problem
Image and sentence matching is challenged by the visual-semantic discrepancy between pixel-level image representations and sentence-level semantic information.
Method
A multi-regional multi-label CNN predicts image concepts, while gated fusion with global context and matched-sentence generation supervision learns their semantic order.
Results
State-of-the-art results are achieved on two public benchmark datasets for image annotation and retrieval.
Takeaways & Limitations
Ablations identify the combination of semantic concepts, extracted context, gated fusion, and generation supervision as the best-performing configuration.
Takeaways & Limitations
The semantic order is difficult to visualize, and future work proposes more accurate concept prediction, end-to-end training, and further investigation for image captioning.
Abstract
from arXiv · showhide
Image and sentence matching has made great progress recently, but it remains challenging due to the large visual-semantic discrepancy. This mainly arises from that the representation of pixel-level image usually lacks of high-level semantic information as in its matched sentence. In this work, we propose a semantic-enhanced image and sentence matching model, which can improve the image representation by learning semantic concepts and then organizing them in a correct semantic order. Given an image, we first use a multi-regional multi-label CNN to predict its semantic concepts, including objects, properties, actions, etc. Then, considering that different orders of semantic concepts lead to diverse semantic meanings, we use a context-gated sentence generation scheme for semantic order learning. It simultaneously uses the image global context containing concept relations as reference and the groundtruth semantic order in the matched sentence as supervision. After obtaining the improved image representation, we learn the sentence representation with a conventional LSTM, and then jointly perform image and sentence matching and sentence generation for model learning. Extensive experiments demonstrate the effectiveness of our learned semantic concepts and order, by achieving the state-of-the-art results on two public benchmark datasets.
1. Introduction
Image and sentence matching measures visual-semantic similarity for cross-modal retrieval, but pixel-level image representations often lack the high-level concepts and ordering expressed in matched sentences. The proposed model addresses this discrepancy by learning semantic concepts, organizing them with image context and sentence-order supervision, and jointly training matching and generation.
- Image and sentence matching measures visual-semantic similarity and supports image annotation and text-based image search.
- High-level objects, properties, and actions are essential for comparison but are difficult to represent from pixel-level images.Global CNN features can tangle these concepts and obscure primary foreground information.
- Incorrect semantic ordering can change the meaning of the same concepts, while multiple plausible incorrect orders make direct order learning difficult.Image global context is proposed as a reference for organizing concepts in a sentence-comparable order.
- Image captioning is an imperfect representation because generated sentences may miss image details, and those errors can affect similarity measurement.The matching score aggregates local similarities in image details.
- The model predicts concepts from multiple image regions, fuses them with global context through gating, and uses matched-sentence order as generation supervision.The predicted concepts include objects, properties, and actions; the context contains spatial relations among concepts.
- The enhanced image representation is matched with an LSTM sentence representation using a structured objective alongside sentence-generation learning.Experiments on image annotation and retrieval across two public datasets achieve state-of-the-art results.
2. Related Work
Prior work learns cross-modal image and sentence representations through ranking, correlation, reconstruction, and recurrent generation objectives. Other approaches improve sentence or image representations with Fisher Vectors, region-to-phrase correspondences, and multimodal or captioning architectures.
- Visual-semantic embedding methods combine CNN image encoders, Skip-Gram or LSTM sentence encoders, and objectives such as ranking or order preservation.
- Deep canonical correlation analysis associates matched images and sentences through high correlation, while related methods enhance sentence representations or model region-to-phrase correspondences.Fisher Vectors, recurrent aggregation, and within-view constraints are among the explored additions.
- Multimodal auto-encoders, multimodal recurrent networks, and long-term recurrent convolutional networks use reconstruction, cross-modal likelihood, perplexity, or captioning architectures for related matching tasks.
3. Semantic-enhanced Image and Sentence Matching
The model extracts image semantic concepts, fuses them with global context to learn their order, and uses sentence generation as supervision alongside image-sentence matching.
- The model comprises sentence representation learning, semantic concept extraction, semantic order learning, and joint matching-generation objectives.
- 3.2. Image Semantic Concept Extraction: Semantic concepts are built from selected nouns, adjectives, verbs, and numbers, with infrequent words and morphological variants normalized to form a vocabulary of K concepts.
- 3.2. Image Semantic Concept Extraction: A multi-label CNN predicts image concepts from selectively extracted regions, sharing parameters across regions and max-pooling their confidence vectors.
- 3.3.1 Global Context as Reference: Global image context provides coarse concept coverage and spatial relations, while a gated fusion unit balances context and concept information into a fused representation.
- 3.3.2 Sentence Generation as Supervision: The fused image representation initializes a generative LSTM, whose predictions are conditioned on previous words, semantic concepts, and initial image context.
- 3.4. Joint Matching and Generation: Joint learning minimizes L = Lmat + λ × Lgen, combining structured matching loss with sentence-generation loss, where λ balances the objectives.
- 3.4. Joint Matching and Generation: At testing time, the model computes image and sentence representations and their cosine similarity without generating a sentence.
4. Experimental Results
Experiments on Flickr30k and MSCOCO evaluate ablations, balancing, state-of-the-art comparisons, and qualitative retrieval behavior. The full model combines semantic concepts, global context, gated fusion, and sentence-generation supervision, achieving strongest reported performance across the evaluations.
- Ablation Models: 10-crop context produces more robust global features than 1-crop context, while pre-generated sentences do not improve matching because they may omit image details.The ablation study also evaluates semantic concepts, generation supervision, scheduled sampling, and shared embeddings.
- Ablation Models: The best ablation is cnp + ctx + gen, combining semantic concepts, 10-cropped context, gated fusion, and sentence-generation supervision; removing concepts or context causes heavy performance drops.The generation objective performs best when its balancing parameter is λ=1, indicating equal weighting with the matching objective.
- State-of-the-art Comparison: On MSCOCO, the proposed model outperforms current state-of-the-art methods by a large margin on all 7 evaluation criteria with either VGGNet or ResNet.On Flickr30k, it is below 2WayNet on R@1 with VGGNet, performs better on the remaining criteria, and achieves the best result with ResNet.
- State-of-the-art Comparison: With 5000 MSCOCO test images, all methods score lower because the larger target set provides more distractors, but the proposed model remains best.Its larger VGGNet improvement than ResNet is attributed to semantic concepts being extracted only with the VGGNet configuration.
- Qualitative Analysis: Qualitative results show that context alone misses or misidentifies concepts, semantic concepts improve retrieval, and semantic-order learning recovers additional image details.The multi-regional multilabel CNN predicts detailed concepts with high confidence, though it incorrectly assigns skate in one complex scene.
5. Conclusions and Future Work
The paper concludes that semantic concepts and their correct order improve image representation for image–sentence matching. It also identifies visualization of learned order and end-to-end concept prediction as areas for future work.
- The proposed model improves image representation by learning semantic concepts and organizing them in the correct semantic order.Its components include a multi-regional multi-label CNN, gated fusion unit, and joint matching and generation learning.
- Systematic component studies demonstrate significant performance improvements for image and sentence matching.
- The learned semantic order is difficult to visualize because it is implied in the image representation.Its effectiveness is instead validated by showing that it helps retrieve more accurate sentences.
- Future work proposes replacing VGGNet with ResNet in the concept-prediction CNN and jointly training it with the rest of the model end to end.