Source-linked AI summary
A Frustratingly Simple Approach for End-to-End Image Captioning
Ziyang Luo, Yadong Xi, Rongsheng Zhang, Jing Ma
TL;DR
Image captioning commonly depends on object detectors requiring costly fine-grained annotations, and detector errors can propagate into captioning. VC-GPT connects pretrained CLIP-ViT and GPT2 with self-ensemble cross-modal fusion, achieving leading performance across three benchmarks without extra object detectors.
Problem
Existing image-captioning systems often require object detectors trained with large, fine-grained datasets, while detector errors can propagate to captioning models.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The framework removes the requirement for object detectors while retaining leading image-captioning performance across the evaluated benchmarks.
Source: §7 · p. 9Increasing image resolution helps only to a certain extent, and larger resolution is not guaranteed to improve performance because it can widen the pretraining–fine-tuning gap.
Source: §6.2 · p. 9
Abstract
from arXiv · showhide
Image Captioning is a fundamental task to join vision and language, concerning about cross-modal understanding and text generation. Recent years witness the emerging attention on image captioning. Most of existing works follow a traditional two-stage training paradigm. Before training the captioning models, an extra object detector is utilized to recognize the objects in the image at first. However, they require sizeable datasets with fine-grained object annotation for training the object detector, which is a daunting task. In addition, the errors of the object detectors are easy to propagate to the following captioning models, degenerating models' performance. To alleviate such defects, we propose a frustratingly simple but highly effective end-to-end image captioning framework, Visual Conditioned GPT (VC-GPT), by connecting the pre-trained visual encoder (CLIP-ViT) and language decoder (GPT2). Different from the vanilla connection method that directly inserts the cross-attention modules into GPT2, we come up with a self-ensemble cross-modal fusion mechanism that comprehensively considers both the single- and cross-modal knowledge. As a result, we do not need extra object detectors for model training. Experimental results conducted on three popular image captioning benchmarks (MSCOCO, Flickr30k and NoCaps) demonstrate that our VC-GPT achieves either the best or the second-best performance across all evaluation metrics over extensive baseline systems.
1 INTRODUCTION
Existing image-captioning systems commonly rely on object detectors trained with costly fine-grained annotations, while detector errors can propagate into captioning. VC-GPT instead connects CLIP-ViT and GPT2 end to end with self-ensemble fusion, achieving leading benchmark performance without extra object detectors.
1 INTRODUCTION
- Traditional captioning pipelines first train an object detector on large, fine-grained recognition datasets, then train captioning models on detected objects.These annotations are time-consuming and are available mainly for popular datasets such as MSCOCO.Source: §1 · p. 1
- VC-GPT removes the need for extra object detectors by using CLIP-ViT as the visual encoder and GPT2 as the language decoder.The framework trains the captioning system end to end using pretrained visual and language models.Source: §1 · pp. 1–2
- Directly inserting randomly initialized cross-attention into GPT2 can harm its pretrained language knowledge because of the gap between language and vision.This motivates a more suitable connection between CLIP-ViT and GPT2.Source: §1 · pp. 1–2
- VC-GPT introduces self-ensemble fusion that combines single-modal and cross-modal knowledge for language generation.The mechanism is designed to connect the visual and language components while preserving both information sources.Source: §1 · p. 2
- VC-GPT achieves the best or second-best performance across all evaluation metrics against traditional two-stage training baselines.The introduction reports this comparison across the evaluated benchmark setting.Source: §1 · p. 2
2 RELATED WORK
Related work has increasingly used pretrained vision-and-language models, but many captioning and pretraining systems still depend on detector-derived visual features. This paper adapts CLIP-ViT and GPT2 for generative image captioning through a self-ensemble fusion mechanism.
2 RELATED WORK
- CLIP-ViT divides images into patch features and enables visual representation without additional object detectors or object annotations.The paper uses only CLIP’s image encoder for image-to-text generation.Source: §2 · p. 2
- GPT2 is a pretrained decoder-only language model that the paper leverages for caption generation.The framework combines GPT2’s language generation ability with CLIP-ViT’s visual representation.Source: §2 · p. 2
- Many neural captioning and vision-language methods rely on object detectors, whose annotations are unavailable or costly for new datasets.The paper adopts CLIP-ViT to avoid this dependence across a wider range of natural images.Source: §2 · p. 2
- METER connects RoBERTa and CLIP-ViT for discriminative tasks, whereas VC-GPT connects GPT2 and CLIP-ViT for generative image captioning.The distinction concerns the target cross-modal task rather than the use of pretrained visual features.Source: §2 · pp. 2–3
3 METHODOLOGY
VC-GPT connects CLIP-ViT and GPT2 for end-to-end captioning, preserving single-modal language knowledge while adding cross-modal fusion. Its self-ensemble mechanism combines single- and cross-modal predictions, and training uses autoregressive language modeling with optional CIDEr optimization.
3.1 Basic Single-Modal Module
- VC-GPT uses CLIP-ViT as visual encoder and GPT2 as language decoder, avoiding extra object detectors in end-to-end image captioning.The visual encoder processes image patches, while GPT2 supplies pretrained language-generation ability.Source: §3 · p. 3, §3.1 · p. 3
3.2 Cross-Modal Fusion Module
- Directly inserting randomly initialized cross-attention into GPT2 can harm pretrained language knowledge through catastrophic forgetting across the vision-language modal gap.The vanilla framework is therefore treated as undesirable for connecting the two modalities.Source: §3.2 · p. 3
- VC-GPT instead adds extra encoder and cross-attention decoder layers without inserting random parameters into GPT2, preserving its integrity while connecting vision and language.The fusion module provides cross-modal predictions while the original decoder remains the single-modal expert.Source: §3.2 · p. 3
- The self-ensemble module combines single-modal and cross-modal logits with a residual connection to use both language-only and cross-modal knowledge.The mechanism is presented as improving generation by combining the two experts’ predictions.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3 Training Objective
- Training predicts each next token from preceding tokens and encoded visual information using autoregressive language modeling, with cross-entropy and optional CIDEr optimization.The latter follows self-critical sequence-training with REINFORCE to optimize CIDEr directly.Source: §3.3 · p. 4
4 EXPERIMENT SETUP
The experiments evaluate VC-GPT on MSCOCO, Flickr30k, and NoCaps against multiple baseline categories using standard captioning metrics. They also include implementation settings and qualitative comparisons with OSCAR on MSCOCO examples.
Default Model Settings
- VC-GPT uses pretrained CLIP-ViT B/16 and GPT2-base with six extra fusion layers, freezing CLIP during pre-training and updating it during fine-tuning.Cross-modal fusion parameters are randomly initialized.Source: §4 · p. 4
Datasets
- The study evaluates image captioning on MSCOCO, Flickr30k, and NoCaps, with NoCaps measuring generalization to unseen object classes.MSCOCO and Flickr30k use Karpathy’s standard train/validation/test splits of 113.2k/5k/5k and 29.8k/1k/1k images, respectively; NoCaps models are trained with MSCOCO data and evaluated on its 4.5k-image validation set.Source: §4 · p. 4
Evaluation Metrics
- Performance is measured with CIDEr, BLEU@4, METEOR, and SPICE using official evaluation procedures.The paper abbreviates these metrics as C, B@4, M, and S.Source: §4 · pp. 4–5
Baseline Systems
- Baselines cover models without cross-modal pre-training, large-scale cross-modal pre-training, and small-scale cross-modal pre-training.The categories differ in their use and scale of image-caption data during captioning-model pre-training.Source: §4 · p. 5
Implementation Details
- Training uses AdamW, mixed precision, 10 pre-training epochs, 30 fine-tuning epochs, and beam search with beam size 5 for evaluation.Experiments run on eight NVIDIA A100 GPUs with stated image-resizing and learning-rate schedules.Source: §4 · p. 5
Qualitative Evaluation
- Qualitative MSCOCO comparisons show VC-GPT and OSCAR captions alongside two reference captions, with blue marking inaccurate words or captions.The examples cover the first four MSCOCO test images.Source: Figure 4 · p. 6, §4 · p. 6
5 EXPERIMENT RESULTS
Across MSCOCO, Flickr30k, and NoCaps, VC-GPT is evaluated against vanilla, end-to-end, and two-stage baselines, with qualitative and cross-attention analyses. The results support its effectiveness, including strong benchmark performance and implicitly learned visual grounding.
5.1 Comparing with Vanilla Framework
- Dropping the self-ensemble module reduces performance on every evaluation metric, with CIDEr decreasing by more than 7 points.Source: §5.1 · p. 6
5.2 Comparing with End-to-End Baseline Systems
- VC-GPT outperforms end-to-end baselines across all evaluation metrics on MSCOCO, including a 2.5-point CIDEr advantage over ViTCAP.Source: §5.2 · p. 7
5.3 Comparing with Two-Stage Training Baseline Systems
- VC-GPT achieves either the best or second-best performance across all evaluation metrics against traditional two-stage systems.Source: §5.3 · p. 7
- On Flickr30k, VC-GPT achieves a CIDEr score of 79.6, surpassing XGPT by more than 8.0 points with less pre-training data.Source: §5.3 · p. 7
- On NoCaps, VC-GPT achieves the best overall performance, with OSCAR’s overall CIDEr score 5 points lower.Source: §5.3 · p. 7
5.4 Qualitative Evaluation
- Qualitative examples show successful image description and motion recognition, while ambiguous object configurations remain difficult for VC-GPT and OSCAR.Source: §5.4 · p. 8
5.5 Cross-Attention Map Visualization
- Cross-attention maps show token-specific attention to corresponding image regions and related objects, indicating implicit visual grounding.Source: Figure 5 · p. 7, §5.5 · p. 8
6 EXTRA EMPIRICAL EXPLORATION
The experiments examine freezing strategies, learning rates, and fine-tuning image resolutions for VC-GPT. Freezing CLIP-ViT alone performs best among the tested freezing settings, while uniform learning rates and moderately increased resolution improve performance under specific conditions.
6 EXTRA EMPIRICAL EXPLORATION
- The section evaluates freezing choices, pre-training learning rates, and fine-tuning image resolutions as training settings for VC-GPT.The exploration covers both cross-modal pre-training and downstream fine-tuning conditions.Source: §6 · p. 8, §6.1 · p. 8
6.1 Freezing CLIP-ViT and/or GPT2 during Cross-modal Pre-training?
- Freezing CLIP-ViT alone improves downstream performance, whereas freezing GPT2 or both single-modal models greatly degrades it.These results identify the framework’s selected freezing strategy as the strongest tested setting.Source: §6.1 · p. 8
6.2 Image Resolution and Learning Rate
- Using the same learning rate for all trainable parameters performs better than assigning a larger rate to the randomly initialized fusion module.The visual encoder is frozen and therefore has no learning rate.Source: §6.2 · p. 8
- Increasing fine-tuning image resolution improves MSCOCO performance to a certain extent.Figure 6 compares different fine-tuning resolutions on the MSCOCO captioning task.Source: §6.2 · p. 8, Figure 6 · p. 8
- Larger fine-tuning resolution is not guaranteed to improve performance because it can increase the gap between pre-training and fine-tuning.Changing resolutions also requires interpolating the vision encoder’s position embeddings.Source: §6.2 · p. 9
7 CONCLUSION
The paper presents VC-GPT as an end-to-end image captioning framework that combines single- and cross-modal knowledge without requiring object detectors. Across MSCOCO, Flickr30k, and NoCaps, it achieves either the best or second-best performance across all evaluation metrics over extensive baselines.
7 CONCLUSION
- VC-GPT achieves the best or second-best performance across all evaluation metrics on three image captioning benchmarks while removing the requirement for object detectors.The evaluated benchmarks are MSCOCO, Flickr30k, and NoCaps.Source: §7 · p. 9