PaperlayerView exact revision on arXiv

Source-linked AI summary

Injecting Semantic Concepts into End-to-End Image Captioning

Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, Zicheng Liu

arXiv:2112.05230v2cs.CVcs.CL

TL;DR

Detector-based captioners rely on regional features, while detector-free vision-language work has focused more on image understanding than caption generation. ViTCAP combines grid representations with semantic concept tokens in an end-to-end transformer captioner, achieving competitive benchmark results against detector-based models.

  • Problem

    Most captioning models rely on object detectors for regional features, while detector-free grid-based work has focused mainly on image understanding and only rarely on generation.

    Source: §1 · pp. 1–2

  • Method

    ViTCAP uses a vision transformer to produce grid representations and a Concept Token Network to predict caption-derived semantic concept tokens for multimodal caption decoding.

    Source: §1 · p. 2

  • Results

    138.1 CIDEr on COCO-caption Karpathy split, 108.6 on Google-CC, and 95.4 on nocaps; ViTCAP surpasses most detector-based models and approaches state of the art.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ViTCAP demonstrates competitive detector-free captioning across challenging datasets, with clear inference-time advantages over detector-based prior arts.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Tremendous progress has been made in recent years in developing better image captioning models, yet most of them rely on a separate object detector to extract regional features. Recent vision-language studies are shifting towards the detector-free trend by leveraging grid representations for more flexible model training and faster inference speed. However, such development is primarily focused on image understanding tasks, and remains less investigated for the caption generation task. In this paper, we are concerned with a better-performing detector-free image captioning model, and propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features. For improved performance, we introduce a novel Concept Token Network (CTN) to predict the semantic concepts and then incorporate them into the end-to-end captioning. In particular, the CTN is built on the basis of a vision transformer and is designed to predict the concept tokens through a classification task, from which the rich semantic information contained greatly benefits the captioning task. Compared with the previous detector-based models, ViTCAP drastically simplifies the architectures and at the same time achieves competitive performance on various challenging image captioning datasets. In particular, ViTCAP reaches 138.1 CIDEr scores on COCO-caption Karpathy-split, 93.8 and 108.6 CIDEr scores on nocaps, and Google-CC captioning datasets, respectively.

1. Introduction

The introduction presents ViTCAP as a detector-free, transformer-based captioner that uses grid features and a Concept Token Network to add semantic concepts. It reports competitive results across captioning benchmarks while targeting the computational and annotation burdens of detector-based models.

1. Introduction

  • Detector-based captioners incur costly regional operations, training inefficiency, high inference latency, and box-annotation constraints, while detector-free generation remains underexplored.A cited detector-free model scored 117.3 CIDEr on COCO, versus 129.3 for VinVL-base.Source: §1 · pp. 1–2
  • The Concept Token Network predicts semantic concepts as tokens using classification trained on pseudo-ground-truth concepts extracted from image captions.The multimodal module decodes from both grid representations and Top-K concept tokens; using detector tags as pseudo-ground-truth yielded no further improvement.Source: §1 · p. 2
  • 138.1 CIDEr on COCO-caption Karpathy split, 108.6 on Google-CC, and 95.4 on nocaps; ViTCAP surpasses most detector-based models and approaches state of the art.It also significantly outperforms existing detector-free captioning models.Source: §1 · p. 2
  • ViTCAP uses a fully transformer architecture with grid representations, avoiding an object detector and intermediate regional operations.The model incorporates semantic concepts through a Concept Token Network.Source: §1 · p. 2
  • Evaluations across multiple captioning datasets report competitive or leading results among detector-based prior arts, with clear inference-time advantages.The paper attributes improved captioning performance to injecting semantic concepts into multimodal fusion.Source: §1 · p. 2

2. Related Work

Related work surveys detector-based captioning and newer efficiency-oriented vision-language approaches, including models that use non-detector image encoders.

2. Related Work

  • Prior captioning work explores new architectures, learning objectives, and large-scale vision-language pretraining, while most models use detectors for regional features.Some studies find strong object detectors necessary, and object tags have been reported as cross-modal anchors.Source: §2 · p. 2
  • Detectors can hinder edge deployment because of their slow inference, motivating efficient vision-language models that optimize detectors or use non-detector encoders.Examples include lightweight detectors and grid-feature encoders.Source: §2 · p. 2
  • Other efficiency efforts use vision-language distillation or transformer pruning, with close performance reported at 50%-70% model sparsity.Source: §2 · p. 2

3. ViTCAP

ViTCAP replaces detector-based regional features with vision-transformer grid representations and adds a Concept Token Network (CTN) to predict semantic concepts for captioning. Its training combines concept classification, masked-token captioning, and teacher-based distillation.

3. ViTCAP

  • ViTCAP uses a detector-free vision transformer and CTN to predict semantic concepts from intermediate image features for multimodal caption generation.The multimodal module receives both concept tokens and final grid representations.Source: §3 · p. 3

3.1. Model Structure

  • The image encoder partitions each image into patches, producing intermediate features for CTN predictions and final grid features for captioning.The final representation is used as grid features; the output of the first M1 blocks feeds the CTN.Source: §3.1 · p. 3
  • The CTN uses the intermediate features and a [CLS] representation to predict top-50 concept tokens from the caption-shared vocabulary.Predicting tokens rather than tags avoids dependence on box annotations and an extra tokenization step.Source: §3.1 · pp. 3–4
  • The shallow multimodal transformer concatenates concept-token embeddings with grid representations and autoregressively generates caption tokens.The seq2seq attention mask lets generated tokens access preceding tokens and visual inputs, but not vice versa.Source: §3.1 · p. 4

3.2. Model Training

  • CTN training predicts concepts through multilabel classification, using concepts extracted from open-form captions or detector-produced tags as pseudo-labels.The authors report better results with caption-extracted concepts and use asymmetric focal loss for the imbalanced concept distribution.Source: §3.2 · p. 4
  • Caption training minimizes negative log-likelihood over masked tokens conditioned on preceding words, grid representations, and concept tokens.The ground-truth set for this loss consists of the masked tokens.Source: §3.2 · p. 4
  • ViTCAP also uses classification distillation over masked-token probabilities from a detector-based captioning teacher, and combines this term with concept and caption losses.The teacher is VinVL; its regional visual features differ from ViTCAP's, so attention-map and hidden-state distillation losses are not directly applicable.Source: §3.2 · p. 4

4. Experiment

The experiments compare ViTCAP with prior captioning models and evaluate it on COCO, Google-CC, and nocaps. Ablations examine semantic-concept sources, model components, distillation, and pre-training.

4.1. Datasets

  • The evaluation covers COCO captions, Google-CC, and nocaps, while pre-training uses 9.9M image-text pairs from four datasets.The pre-training corpus comprises 4.1M independent images; the COCO Karpathy split uses 113k, 5k, and 5k images for training, validation, and testing.Source: §4.1 · p. 5

4.2. Implementation Details

  • ViTCAP uses a ViT-base encoder with eight stem blocks and four CTN blocks, and trains concept classification before joint captioning to avoid cold start.The joint captioning stage initializes from the concept-classification model; the feature extractor is initialized from the CTN branch.Source: §4.2 · p. 5

4.3. Main Results

  • Without VLP, ViTCAP reaches CIDEr 121.8 on COCO-caption Karpathy split, surpassing most detector-based methods despite using a vanilla transformer architecture.The comparison uses VinVL tags as concept sources and omits knowledge distillation.Source: §4.3 · p. 6, Table 1 · p. 6
  • With large-scale pre-training, ViTCAP scores CIDEr 125.2 vs. 123.7 for OSCARb, exceeds ViLT-CAP by 11.7 and E2E-VLP by 7.9, and trails VinVL by 2.3 after CIDEr optimization.These COCO-caption Karpathy-split comparisons are reported for ViTCAP with VLP; the VinVL comparison follows CIDEr optimization.Source: §4.3 · p. 6, Table 1 · p. 6

4.4. Ablative Study

  • On COCO, caption-keyword concepts improve CIDEr from 114.8 to 120.9, all caption tokens improve it further, and VinVL tags score 119.7 versus 117.4 for BUTD.The authors also report that combining VinVL tags in the first stage with caption tokens in the second is viable when captions are unavailable.Source: §4.4 · p. 7, Table 2 · p. 7
  • On COCO-caption, the baseline scores 114.8 CIDEr; explicit CTN-predicted tags improve it by 2.3 CIDEr and 1.3 BLEU@4, reaching 117.4 versus 117.1 with VinVL detector tags.Source: §4.4 · p. 7
  • ViTCAP reaches CIDEr 108.6 on Google-CC and 98.7 and 93.8 on nocaps in-domain and out-of-domain splits, respectively, exceeding VIVO by 6.5 and 5.5.The authors report promising results across all three nocaps splits and attribute generalization partly to the broader concept vocabulary from open-form captions.Source: §4.4 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5. Conclusion

ViTCAP is a detector-free, fully transformer-based captioning model that uses grid representations and CTN-learned semantic concepts. Experiments show competitive performance across challenging benchmarks.

5. Conclusion

  • ViTCAP performs end-to-end captioning with grid representations, without intermediate regional operations.Its full architecture is transformer-based.Source: §5 · p. 8
  • The Concept Token Network learns broad semantic concepts and encodes them as tokens that benefit captioning.Source: §5 · p. 8
  • Across challenging captioning benchmarks, ViTCAP achieves competitive performance, approaching most detector-based models.Source: §5 · p. 8

6. Pre-training VL Corpus

ViTCAP pre-training combines several image-caption datasets into one corpus, with duplicate images removed against the COCO Karpathy test split for fair comparison.

6. Pre-training VL Corpus

  • The pre-training corpus aggregates COCO, Conceptual Caption, SBU Captions, and Visual Genome.Source: §6 · p. 9
  • The aggregated corpus contains 4.2 million images and 9.9M captions.Source: §6 · p. 9
  • Images shared with the COCO Karpathy testing splits are de-duplicated from pre-training for fair comparisons.Source: §6 · p. 9

7. Ablative Studies

The ablations examine concept sources, training choices, and architecture variants alongside inference cost. They report that ViTCAP uses substantially fewer FLOPs than a prevailing VL model and identify several useful concept-training choices.

Inference speed and parameters

  • 97G FLOPs for ViTCAP versus 1,025G for VinVL, with on-par parameters, amounts to about 10% of VinVL’s FLOPs.Source: Figure 4 · p. 9, §7 · p. 9

More About Concept Sources

  • Caption-derived concepts can be noisy or incomplete, while detector tags are less noisy but restricted to a fixed vocabulary.Open-form captions offer broader concepts, but their extracted concepts may cover only part of an image.Source: §7 · p. 9

Concept Classification Training

  • 117.1 CIDEr is reported with focal loss; concept-classification pre-training further improves performance, and Tag+Init outperforms Init.The study examines binary cross-entropy and focal loss for imbalanced concept classification.Source: §7 · pp. 9–10, Table 7 · p. 10, Figure 5 · p. 10

Representing Concepts as Tokens

  • An independent tokenizer yields only 0.4 higher CIDEr, so the authors share the captioning tokenizer to avoid additional parameters.Source: Table 8 · p. 10, §7 · p. 10

Training paradigm

  • The training diagram presents detector-based teacher assistance as distillation, while the CTN learns semantic concept tokens for captioning.Source: Figure 6 · p. 11

Training with concept tokens

  • Training with predicted concepts gives optimal results, which the authors attribute partly to the pre-trained CTN producing reasonable concepts during captioning fine-tuning.Source: §7 · p. 11, Table 9 · p. 11

ViTCAP Architecture

  • Using fewer shared stem blocks can improve performance but increases model size; the two-tower design is used for large-scale pre-training.Source: §7 · p. 11, Table 10 · p. 11

More Evaluations

  • ViTCAP preserves semantic performance and term descriptiveness under the SMURF evaluation.The supplied table caption identifies the comparison as using ViTCAP without pre-training.Source: §7 · p. 11, Table 11 · p. 12

8. Discussions

ViTCAP’s transformer attention maps are associated with predicted concepts, but may highlight only parts of objects. The discussion also contrasts broad caption-derived concepts with detector vocabulary and describes the scope of its distillation setup.

Can ViTCAP Ground Concepts?

  • Different transformer layers focus on different concepts, while averaged attention maps cover broader holistic regions.The visualizations include attention maps from selected blocks and averages across early and all blocks.Source: §8 · p. 12, Figure 7 · p. 13, Figure 8 · p. 14
  • Attention can highlight only parts of an object, such as a traffic light or tie, rather than its clear borders.Source: §8 · p. 12

VL Distillation Schema

  • The teacher-distillation scheme is applied only to ViTCAP with VL pre-training, because the teacher contains knowledge from large-scale pre-training.The authors describe comparison with methods without VL pre-training as unfair.Source: §8 · p. 12

Detector Tags vs. Caption Extracted Concepts

  • Caption-derived concepts have broader vocabulary than detector tags, but noisy web captions may cover only part of an image’s content.Source: §8 · p. 12

Related papers

Loading 2112.05230v2…