Source-linked AI summary
GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features
Van-Quang Nguyen, Masanori Suganuma, Takayuki Okatani
TL;DR
Image captioning needs visual features that capture both objects and scene context, but region-based approaches can suffer from missed context, detection errors, costly computation, and difficult joint training. GRIT integrates grid and region features in a Transformer-only architecture using DETR-based extraction and a dual-feature caption generator. It achieves state-of-the-art or competitive performance across several benchmarks while improving inference efficiency.
Problem
Region features provide object-level information but can lack context, suffer detection errors, incur high computational cost, and hinder end-to-end training.
Method
GRIT integrates grid and region features using a Transformer-only architecture with DETR-based extraction and cross-attention to caption words.
Results
GRIT outperforms methods without vision-language pretraining on COCO and performs at least on par with SimVLMhuge, while surpassing competing methods on additional benchmarks.
Takeaways & Limitations
Integrating complementary grid and region features in a unified Transformer architecture yields richer visual representations for image captioning.
Abstract
from arXiv · showhide
Current state-of-the-art methods for image captioning employ region-based features, as they provide object-level information that is essential to describe the content of images; they are usually extracted by an object detector such as Faster R-CNN. However, they have several issues, such as lack of contextual information, the risk of inaccurate detection, and the high computational cost. The first two could be resolved by additionally using grid-based features. However, how to extract and fuse these two types of features is uncharted. This paper proposes a Transformer-only neural architecture, dubbed GRIT (Grid- and Region-based Image captioning Transformer), that effectively utilizes the two visual features to generate better captions. GRIT replaces the CNN-based detector employed in previous methods with a DETR-based one, making it computationally faster. Moreover, its monolithic design consisting only of Transformers enables end-to-end training of the model. This innovative design and the integration of the dual visual features bring about significant performance improvement. The experimental results on several image captioning benchmarks show that GRIT outperforms previous methods in inference accuracy and speed.
1 Introduction
GRIT addresses limitations of region-only image captioning by integrating complementary grid and region features in a Transformer-only architecture. It uses DETR-based feature extraction to support faster inference and end-to-end training.
- Motivation: Image captioning generates a semantic scene description from an image, making effective visual feature extraction central to performance.Existing systems generally extract visual features before generating the description.
- Motivation: Region features provide object-level information but can miss contextual relations, detect objects inaccurately, and require costly CNN-based computation.These limitations are especially associated with detectors such as Faster R-CNN.
- Motivation: Grid features complement region features by representing whole-image context and avoiding erroneous object detection, although they lack object-level information.The paper therefore treats the two feature types as complementary representations.
- Approach: GRIT uses Deformable DETR with a Swin Transformer backbone to extract region and grid features, replacing CNN-based detection and updating grid features with self-attention.The design targets both lower computational cost and richer contextual representation.
- Approach: Its lightweight caption generator applies cross-attention from both visual feature types to caption words in an autoregressive Transformer.The caption generator is integrated with the dual-feature extraction pipeline.
- Results: GRIT establishes a new COCO state of the art, outperforming methods without vision-language pretraining and matching at least SimVLMhuge with such pretraining.The comparison uses the Karpathy test split and offline evaluation.
2 Related Work
Prior image-captioning work moved from holistic CNN representations toward grid, region, and Transformer-based visual features. These approaches differ in feature granularity, visual encoding, and how Transformers are used across captioning pipelines.
- Visual representations: Early methods used compact holistic CNN features, but their global representation suffered information loss and insufficient granularity.Later work adopted finer-grained grid-based features to represent images.
- Transformer approaches: Transformers have been used for visual encoding, region-feature extraction, and caption generation, including self-attention and specialized image-captioning variants.DETR and its variants also extended Transformer architectures to object detection.
3 Grid- and Region-based Image captioning Transformer
GRIT is a Transformer-based image-captioning architecture that extracts complementary region and grid features, then fuses them through specialized cross-attention in an autoregressive caption generator. It replaces conventional CNN-based detection components with a DETR-based object detector and supports end-to-end training.
- Overall architecture: GRIT consists of a dual-feature extractor and a caption generator, both built around Transformer components.The architecture extracts visual features first and generates captions from them in a separate model component.
- Visual feature extraction: A Swin Transformer backbone produces four multi-scale feature maps used by the object detector and grid-feature network.The maps range from H/8 × W/8 to H/64 × W/64 in spatial resolution.
- Generating region features: GRIT uses a Deformable DETR decoder with learnable object queries to produce region features R from the backbone’s multi-scale features.The final object-query representations are forwarded to the caption generator as region features.
- Generating grid features: A Transformer network updates the backbone’s final feature map to produce contextual grid features G.The grid-feature network operates on the last multi-scale feature map after a learnable linear projection.
- Caption generation: The caption generator autoregressively predicts words using masked self-attention followed by cross-attention with the dual visual features.Its output is projected to vocabulary-sized logits for next-word prediction.
- Cross-attention designs: GRIT evaluates concatenated, sequential, and parallel cross-attention designs for combining region and grid features.Concatenated attention joins the visual features, sequential attention applies two attention modules in order, and parallel attention uses independent mechanisms; sequential order may affect performance.
4 Experiments
Experiments evaluate GRIT through ablations, COCO comparisons, cross-dataset tests, and inference-speed measurements. The results show benefits from broader visual pretraining, integrated visual features, end-to-end training, strong benchmark performance, and faster inference.
- Ablation Studies: Using more pretraining datasets leads to better image captioning performance.The comparison includes ImageNet-only grid features, Visual Genome detector pretraining, and pretraining on four datasets.
- Ablation Studies: Performance increases with 50, 100, and 150 object queries, then saturates while computational cost and false detection increase.The ablation identifies a trade-off between adding region features and increasing computation and detection errors.
- Ablation Studies: End-to-end training improves CIDEr from 139.6 to 144.3 with little sacrifice of B@4.The comparison contrasts end-to-end training with training that leaves the vision backbone fixed.
- Ablation Studies: Fusing grid and region features achieves the highest performance, while region features alone outperform grid features alone.The fusion comparison evaluates concatenated cross-attention and single-feature variants.
- COCO Evaluation: GRIT achieves the best scores for all metrics in online COCO evaluation, and its single model outperforms published ensemble methods.The online evaluation uses five and 40 reference captions per image.
- Cross-Dataset Evaluation: GRIT outperforms existing methods by a large margin on ArtEmis and surpasses other methods on both in-domain and out-of-domain nocaps images.For nocaps, the model is trained on COCO and evaluated zero-shot on images containing novel objects and unseen vocabularies.
- Computational Efficiency: GRIT reduces feature-extraction time by a factor of 10 and reaches about 32ms per image with minibatch size ≥32.Inference measurements use a V100 GPU, caption length 20, and beam size five.
5 Summary and Conclusion
GRIT combines dual visual features in a Transformer-based architecture and uses DETR-based detection to address limitations of prior region-feature systems. Experiments report improved image-captioning accuracy and speed.
- GRIT integrates region and grid features to extract richer visual information from input images.The architecture uses a Transformer-based caption generator with cross-attention over both feature types.
- A Swin Transformer backbone and DETR-based detector address the training and computational issues associated with CNN-based detectors.The DETR-based design avoids the limitations that previously prevented end-to-end training.
- GRIT outperforms published methods by a large margin in inference accuracy and speed.
A.1 Object Detection Datasets
The object-detection pretraining setup combines four datasets and specifies detector architecture, initialization, and bounding-box loss components.
- The unified object-detection pretraining corpus combines Visual Genome, COCO, OpenImages, and Objects365, covering 1848 categories.COCO stuff annotations are excluded from the corpus.
- The detector uses 150 queries, four sampling points, hidden dimension d = 512, and Swin-Base weights pretrained on ImageNet21K.
- Bounding-box loss is a weighted combination of L1 box distance and generalized intersection-over-union loss.The stated weights are α_l1 = 5 and α_iou = 2.
A.3 Object Detection Results
GRIT's object detector achieves performance comparable to or higher than established detectors on COCO and Visual Genome, while the surrounding architecture includes task-specific token handling.
- GRIT's object detector attains comparable or higher performance than BUTD and VinVL on COCO and Visual Genome when pretrained on similar datasets.
- Table 6 reports object-detection performance on the COCO validation split and Visual Genome test split, with 4DS denoting four object-detection datasets.
- The grid-feature network prepends a class token for predicting emotion categories in an emotion-grounded ArtEmis model.
- During caption training and inference, GRIT uses ⟨sos⟩ to start captions and ⟨eos⟩ to terminate them.
SPICE Sub-category and CLIPscore Metrics
The SPICE sub-category analysis evaluates image-captioning performance across object, attribute, and relationship descriptions, with GRIT improving over the compared region-based methods.
- GRIT achieves better SPICE F-scores across all reported sub-categories than Up-Down, vanilla Transformer, and M2 Transformer.
- The sub-category results show significant improvement in identifying and counting objects, describing attributes, and modeling relationships between objects.
- The table also reports CLIP scores and shows consistent improvement over the compared method.
B.2 Image Captioning on the ArtEmis dataset
The ArtEmis dataset contains 80,031 images and 454,684 captions annotated across eight emotion categories. The authors also evaluate an emotion-grounded model trained jointly for emotion prediction and caption generation.
- 80,031 unique images are divided into training, validation, and test splits using 85%, 5%, and 10% ratios.
- 454,684 captions are annotated with labels from eight unique emotion categories.
- The emotion-grounded model projects the updated class embedding into an 8-dimensional vector for emotion prediction.
- Training minimizes the combined emotion-prediction and caption-generation losses.
- ArtEmis test results include emotion-grounded models, but ground-truth emotion labels are unavailable during inference.
B.3 Image Captioning on the nocaps Dataset
The paper reports nocaps validation results across in-domain, near-domain, and out-of-domain settings.
- The nocaps validation evaluation covers in-domain, near-domain, and out-of-domain domains.
- Results are reported for different models on the nocaps validation split.
- The domain-specific results are presented in Table 9.
Full results
The full results compare GRIT with region-based methods in feature extraction, caption generation, inference speed, and qualitative caption quality. The passages attribute major timing differences to detector and decoder computation.
- Inference evaluation: Inference-time measurements compare GRIT, VinVL, and M2 Transformer on the same Tesla V100-SXM2 16GB system with preprocessing excluded.
- Feature extraction: VinVL and M2 Transformer spend considerable feature-extraction time on CNN backbones with high-resolution inputs and regional operations.
- Feature extraction: VinVL also uses class-agnostic NMS operations during region-feature extraction.
- Caption generation: M2 Transformer and GRIT use three-layer, 512-dimensional caption decoders, whereas VinVLlarge uses 24 layers with dimension 1024.
- Caption generation: All compared methods use beam search with beam size 5 and maximum caption length 20.
- Qualitative results: GRIT captions are qualitatively better than M2 Transformer captions for detecting and counting objects and describing relationships.