Comprehending and Ordering Semantics for Image Captioning

Yehao Li; Yingwei Pan; Ting Yao; Tao Mei

Source-linked AI summary

Comprehending and Ordering Semantics for Image Captioning

Yehao Li, Yingwei Pan, Ting Yao, Tao Mei

arXiv:2206.06930v1Jun 14, 2022cs.CVcs.AIcs.CLcs.MM

TL;DR

Image captioning needs rich, visually grounded semantics arranged in linguistic order, but existing methods under-exploit semantic ordering and rely on constrained detectors or language priors. COS-Net retrieves sentence-derived cues, filters and enriches them, ranks them linguistically, and integrates them with visual tokens for generation. It demonstrates effectiveness on COCO, while its learned descriptions can reflect biases in the training data.

Problem
Image captioning must cover salient image semantics in linguistic order, whereas detector-based methods have limited label-defined semantics and decoder language priors can contribute to object hallucination.
Method
COS-Net uses CLIP to retrieve sentence-derived semantic cues, filters and enriches them with a semantic comprehender, ranks them linguistically, and integrates them with visual tokens for decoding.
Results
Extensive experiments on COCO demonstrate the effectiveness of COS-Net, and qualitative results show more relevant semantic coverage than Transformer † and Up-Down † baselines.
Takeaways & Limitations
The paper supports jointly using enriched semantic comprehension and learnable linguistic ordering as additional guidance for image-caption generation.
Takeaways & Limitations
Because COS-Net learns from training-data statistics, its descriptions can reflect dataset biases and produce negative societal impacts.

Abstract

from arXiv · show

Comprehending the rich semantics in an image and ordering them in linguistic order are essential to compose a visually-grounded and linguistically coherent description for image captioning. Modern techniques commonly capitalize on a pre-trained object detector/classifier to mine the semantics in an image, while leaving the inherent linguistic ordering of semantics under-exploited. In this paper, we propose a new recipe of Transformer-style structure, namely Comprehending and Ordering Semantics Networks (COS-Net), that novelly unifies an enriched semantic comprehending and a learnable semantic ordering processes into a single architecture. Technically, we initially utilize a cross-modal retrieval model to search the relevant sentences of each image, and all words in the searched sentences are taken as primary semantic cues. Next, a novel semantic comprehender is devised to filter out the irrelevant semantic words in primary semantic cues, and meanwhile infer the missing relevant semantic words visually grounded in the image. After that, we feed all the screened and enriched semantic words into a semantic ranker, which learns to allocate all semantic words in linguistic order as humans. Such sequence of ordered semantic words are further integrated with visual tokens of images to trigger sentence generation. Empirical evidences show that COS-Net clearly surpasses the state-of-the-art approaches on COCO and achieves to-date the best CIDEr score of 141.1% on Karpathy test split. Source code is available at \url{https://github.com/YehLi/xmodaler/tree/master/configs/image_caption/cosnet}.

1. Introduction

Image captioning requires both rich visual-semantic coverage and linguistic ordering, but existing detector-based methods and language-model decoders leave limitations in these areas. COS-Net unifies semantic comprehending and ordering, and experiments on COCO demonstrate its effectiveness.

1. Introduction: Pre-trained detectors and classifiers capture fine-grained semantics but are constrained by predefined semantic labels and separately optimized from the sentence decoder.These limitations hinder adaptive emphasis on salient semantics worthy of mention.
1. Introduction: CLIP retrieves semantically similar sentences that provide richer semantic cues, although retrieval also introduces irrelevant words.Figure 1 contrasts semantics from an object detector, CLIP retrieval, and the proposed comprehender and ranker.
1. Introduction: The semantic comprehender filters irrelevant words and infers missing relevant words, such as replacing “horse” with visually grounded “cow” and “rides” in the illustrated case.This produces an enriched and more accurate semantic understanding.
1. Introduction: The semantic ranker orders refined semantic words as an explicit skeleton that guides relevant word generation at each decoding timestep.This addresses decoder reliance on language priors and the associated risk of object hallucination.
1. Introduction: COS-Net jointly comprehends and orders image semantics in a unified architecture to improve visually grounded, linguistically coherent caption generation.Its semantic comprehender enriches relevant semantic words, while its ranker arranges them in linguistic order before decoding.
1. Introduction: Extensive COCO experiments demonstrate the effectiveness of COS-Net.The introduction presents this as the empirical validation of jointly comprehending and ordering semantics.

2. Related Work

Prior image-captioning systems use RNN- or Transformer-based encoder-decoder schemes, while COS-Net extends Transformer-style designs with semantic retrieval, comprehending, and ordering.

2. Related Work: RNN-based encoder-decoder schemes became a standard recipe for image captioning, encoding visual content with CNNs and decoding descriptions with RNNs.Earlier image-captioning attempts directly applied this sequence-modeling framework.
2. Related Work: Transformer-based image captioning strengthens visual encoding and vision-language interaction through self-attention or cross-attention mechanisms.These approaches emerged after Transformer advances in NLP.
2. Related Work: COS-Net is a Transformer-based encoder-decoder scheme whose visual encoder, sentence decoder, and semantic comprehender use Transformer-style structure.Its architecture integrates semantic comprehending and ordering beyond the conventional scheme.
2. Related Work: Compared with CLIP-ViL [29], COS-Net uses CLIP to retrieve richer sentence-derived semantic cues and then refines and linguistically orders them.The semantic comprehender removes irrelevant words and infers missing relevant ones before the semantic ranker allocates linguistic positions.

3. Our Approach: COS-Net

COS-Net unifies visual encoding, semantic comprehending, semantic ordering, and sentence decoding in a Transformer-style architecture for image captioning. It retrieves candidate semantics, filters and enriches them using visual tokens, dynamically orders them, and integrates them with visual features for generation.

3. Our Approach: COS-Net: COS-Net unifies semantic comprehending and ordering in a single Transformer-style architecture for image captioning.The architecture integrates visual encoding, semantic processing, semantic ranking, and sentence decoding.
3.1. Visual Content Encoding: The visual encoder transforms CLIP global and grid features into contextual visual tokens through stacked Transformer blocks and multi-head self-attention.Global features from the Transformer blocks are concatenated into a holistic feature and integrated with encoded grid features.
3.2. Semantic Comprehending: CLIP retrieves semantically relevant training captions by image-text cosine similarity, whose non-stopword terms form the primary semantic cues.The retrieved captions provide candidate words from visually similar images, while potentially introducing irrelevant terms.
3.2. Semantic Comprehending: The semantic comprehender uses Transformer blocks with self-attention and visual cross-attention to screen irrelevant cues and reconstruct missing relevant semantic words.It formulates semantic screening and enrichment as set prediction, using primary cues, parametric queries, and visual tokens.
3.3. Semantic Ordering: The semantic ranker dynamically infers each semantic word’s linguistic position using attention over learnable position encodings instead of fixed positional assignments.This produces position-aware semantic tokens that are combined with enriched visual tokens in the Transformer decoder for sentence generation.

4. Experiments

Experiments evaluate COS-Net on COCO using ablations, standard and online test splits, and a robust split for object-hallucination analysis. COS-Net consistently outperforms compared methods, reaches 141.1% CIDEr on the single-model Karpathy split, and reduces hallucination metrics.

Dataset and Experimental Settings: The experiments use COCO’s standard Karpathy split and a robust split whose caption object pairs do not overlap across training, validation, and testing.COCO contains more than 120,000 images, each paired with five human-annotated sentences.
Ablation Study: Ablations progressively improve performance when adding cross-modal retrieval, irrelevant-semantic filtering, missing-semantic inference, and semantic ranking to the base model.The ablation results are reported without self-critical sequence training on the COCO Karpathy test split.
Comparisons with State-of-the-Art: 141.1% CIDEr gives COS-Net a 3.9% absolute gain over X-Transformer † at 137.2% on the single-model Karpathy test split.The comparison uses CLIP grid features with a ResNet-101 backbone for the upgraded variants.
Comparisons with State-of-the-Art: 7.7% absolute CIDEr improvement separates ensemble COS-Net from ensemble X-Transformer on the Karpathy test split.The ensemble setup averages four models trained with different random seeds.
Comparisons with State-of-the-Art: COS-Net surpasses state-of-the-art approaches across all reported metrics on the official online test split.The ensemble COS-Net is evaluated with both 5 reference captions and 40 reference captions.
Comparisons with State-of-the-Art: Lower CHs and CHi scores on the robust split indicate that COS-Net is more robust by alleviating object hallucination.The analysis compares typical sentence metrics with image-relevance metrics against standard methods and CLIP-feature variants.

5. Conclusion and Discussion

COS-Net unifies enriched semantic comprehending with learnable semantic ordering in a Transformer-style architecture for image captioning. Qualitative examples show more relevant semantic words than two baselines, while the method’s training data can reflect societal biases.

Conclusion and Discussion: COS-Net captures more relevant semantic words than Transformer and Up-Down in qualitative examples, including smoke, multiple dogs, posing men, sheep eating hay, and a motorcycle.The examples pair ordered semantic words with generated captions and ground-truth descriptions.
Conclusion and Discussion: The qualitative comparison covers COS-Net, Transformer, Up-Down, and human-annotated ground-truth descriptions.Figure 3 presents the outputs alongside the corresponding ground-truth descriptions.
Conclusion and Discussion: COS-Net unifies enriched semantic comprehending and learnable semantic ordering in one architecture for image captioning.A CLIP-based retrieval model supplies primary semantic cues; the comprehender filters irrelevant words and infers missing relevant ones, while the ranker orders them linguistically.
Conclusion and Discussion: COS-Net may reflect biases rooted in its training data, potentially resulting in negative societal impacts.The paper states that further research is needed to address this broader-impact issue.