Source-linked AI summary
Image Captioning: Transforming Objects into Words
Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares
TL;DR
Image captioning encoders based on detected regions generally omit spatial relationships between objects, although those relationships can be important for interpreting scenes. The paper introduces the Object Relation Transformer, which adds geometric attention to the Transformer encoder. On MS-COCO, it reports improved captioning metrics and qualitative gains in spatial awareness, while leaving decoder-side geometric attention for future work.
Problem
Detection-based image captioning encoders do not utilize spatial relationships between detected objects, such as relative position and size, despite their importance for understanding image content.
Method
The Object Relation Transformer incorporates detected-object spatial relationships through geometric attention in the Transformer encoder within an encoder-decoder captioning architecture.
Results
6.8% relative improvement over the Up-Down baseline, with state-of-the-art CIDEr-D, SPICE, METEOR, and BLEU-4 results on the reported MS-COCO test metrics.
Takeaways & Limitations
Spatial relationship information benefits Transformer image captioning and can yield captions demonstrating better spatial awareness.
Takeaways & Limitations
The model incorporates geometric information only during encoding, not in decoder cross-attention between objects and words.
Abstract
from arXiv · showhide
Image captioning models typically follow an encoder-decoder architecture which uses abstract image feature vectors as input to the encoder. One of the most successful algorithms uses feature vectors extracted from the region proposals obtained from an object detector. In this work we introduce the Object Relation Transformer, that builds upon this approach by explicitly incorporating information about the spatial relationship between input detected objects through geometric attention. Quantitative and qualitative results demonstrate the importance of such geometric attention for image captioning, leading to improvements on all common captioning metrics on the MS-COCO dataset.
1 Introduction
Image captioning commonly encodes detected image regions but does not use their spatial relationships. The paper introduces the Object Relation Transformer, which adds geometric attention and evaluates its usefulness quantitatively and qualitatively.
- Motivation: Image captioning systems commonly use encoder-decoder architectures with image features from CNNs, sampled regions, or object-detector proposals.Detection-based encoders represent the state of the art described in the introduction.
- Motivation: Spatial relationships between detected objects, including relative position and size, are often omitted despite helping distinguish visually different scene descriptions.The introduction gives examples involving a girl and a horse, and a woman with a guitar or ukulele.
- Proposed approach: The Object Relation Transformer incorporates spatial relationships between detected objects into a Transformer encoder-decoder through geometric attention.Its object relation module is incorporated within the Transformer encoder.
- Evaluation: The paper quantitatively evaluates geometric attention through baseline comparison and ablation on the MS-COCO dataset.The study compares against established captioning baselines and examines the contribution of geometric attention.
- Evaluation: Geometric attention qualitatively produces captions with enhanced spatial awareness.Figure 1 illustrates attention linking a chair to another chair, the beach, and an umbrella.
2 Related Work
Earlier captioning methods used global or region-based visual representations, while attention and graph approaches added localization or explicit relationships with different limitations. The Object Relation Transformer instead weights object pairs using their continuous spatial relation information.
- Object- and region-based representations: Early image captioning models encoded a whole image as one feature vector, omitting object-level spatial relationships.Region-based work extracted multiple detector regions but generated separate captions without modeling relationships between detected objects.
- Attention-based captioning: Attention-based methods sought to ground predicted words in image regions, but convolutional attention often had limited or weakly meaningful spatial localization.Bottom-up top-down attention improved object-based visual attention using Faster R-CNN proposals and an LSTM.
- Graph-based relationships: Graph-based captioning introduced semantic and spatial relationship graphs, with spatial relations classified into 11 categories such as inside, cover, and overlap.This approach added global context through separate graph convolutional networks.
- Transformer-based captioning: Transformer-based captioning explored global image features and uniformly sampled 8x8 image partitions rather than object-pair spatial relations.The proposed approach extends this line by encoding how two detected objects are spatially related and weighting them accordingly.
3 Proposed Approach
The proposed Object Relation Transformer extends Transformer-based image captioning by incorporating detected-object geometry into encoder attention. It combines appearance-based attention with learned information about relative object position and size.
- Architecture: The model uses object-detector appearance and geometry features as inputs to a Transformer encoder-decoder that generates caption words.The object detector supplies the visual tokens, while the Transformer decoder produces the output caption sequence.
- Architecture: Each image feature vector is projected from 2048 dimensions to dmodel = 512 before entering the encoder.The embedding applies a fully connected layer, ReLU, and dropout.
- Standard Transformer: Each encoder layer applies 8-head self-attention followed by a point-wise feed-forward network, with skip-connections and layer normalization.Queries, keys, and values are learned projections of the input token matrix.
- Geometric Attention: The model modifies appearance attention by multiplying attention between objects by a learned function of their relative position and size.Relative geometry is computed from bounding-box center coordinates, widths, and heights, then transformed into geometric attention weights.
- Geometric Attention: The resulting geometric weights are combined with appearance weights to form the attention matrix used to compute each head's output.The combined weights replace the standard attention weights in the self-attention computation.
4 Implementation Details
The experiments train the proposed model with cross-entropy pretraining followed by self-critical reinforcement learning, and report a fair beam-size-5 comparison in Table 1.
- Training: The best model is pretrained for 30 epochs with softmax cross-entropy and then trained for 30 additional epochs using self-critical reinforcement learning optimized for CIDEr-D.Training uses ADAM, 20,000 warmup steps, and batch size 10.
- Evaluation: Table 1 reports the Object Relation Transformer after self-critical training with beam search using beam size 5.The table compares the model with existing state-of-the-art approaches.
- Evaluation: The best-performing models in Sections 5.3–5.6 use beam size 2, while Table 1 uses beam size 5 for consistency with reported literature results.The differing beam sizes distinguish internal evaluations from the headline comparison.
5 Experimental Evaluation
The evaluation compares the Object Relation Transformer with established captioning systems and Transformer variants on MS-COCO. Results show gains from geometric attention in benchmark metrics, especially relationship and counting measures, with qualitative improvements in spatial awareness and counting.
- Experimental setup: The models were trained and evaluated on MS-COCO 2014 using CIDEr-D, SPICE, BLEU, METEOR, and ROUGE-L across Karpathy validation and test splits.The dataset contains 113K training images and 5K-image validation and test sets.
- Comparative analysis: 6.8% relative improvement over the Up-Down baseline established the Object Relation Transformer as state-of-the-art for CIDEr-D, SPICE, METEOR, and BLEU-4.The comparison used self-critical training optimized for CIDEr-D and beam search with beam size 5.
- Positional encoding: Geometric attention was compared with object ordering by size, left-to-right position, and top-to-bottom position as alternatives to Transformer positional encoding.The proposed mechanism directly models object geometry rather than imposing an ordering.
- Ablation study: The ablation study found small METEOR gains but significant improvements in CIDEr-D and BLEU metrics after adding geometric attention.Overall, the largest improvements occurred in CIDEr-D and BLEU-4.
- Geometric improvement: The Object Relation Transformer scored higher than the Standard Transformer on every metric, with statistically significant gains in CIDEr-D, BLEU-1, ROUGE-L, SPICE Relation, and SPICE Count.SPICE Count increased significantly from 11.30 to 17.51, although captioning systems still lag humans on counting.
- Qualitative analysis: Qualitative examples showed improved captions for object relationships and counts, including more accurate descriptions of object numbers and spatial configurations.Table 6 focuses on SPICE Relation improvements, while Table 7 focuses on SPICE Count improvements.
6 Conclusion
The paper presents the Object Relation Transformer, which encodes 2D position and size relationships among detected objects for image captioning. On MS-COCO, the method benefits from spatial relationship information and produces captions with better spatial awareness, while geometric information is currently limited to the encoder.
- Conclusion: The Object Relation Transformer modifies the conventional Transformer to encode 2D position and size relationships between detected objects.It builds upon the bottom-up and top-down image captioning approach.
- Conclusion: MS-COCO results show that incorporating spatial relationship information benefits the Transformer, especially on relevant SPICE sub-metrics.Qualitative examples also demonstrate better spatial awareness.
- Limitations and future work: The current model uses geometric information only in the encoder phase, leaving decoder cross-attention between objects and words for future work.The authors expect this extension to improve performance and interpretability.