Source-linked AI summary
Exploring Visual Relationship for Image Captioning
Ting Yao, Yingwei Pan, Yehao Li, Tao Mei
TL;DR
Image captioning lacked evidence that modeling object relationships improves image description generation. GCN-LSTM integrates semantic and spatial object graphs into an attention-based LSTM encoder-decoder, and achieves state-of-the-art COCO performance, including CIDEr-D of 128.7%.
Problem
Image captioning had not fully established how visual relationships between objects should be leveraged to support image description generation.
Method
GCN-LSTM builds semantic and spatial graphs over detected regions, refines region representations with graph convolutions, and uses attention-based LSTM decoders for sentence generation.
Results
GCN-LSTM achieves new state-of-the-art performance on the COCO image-captioning dataset, reaching CIDEr-D 128.7% when optimized with CIDEr-D score.
Takeaways & Limitations
Modeling semantic and spatial object relationships provides relation-aware region representations within the captioning framework.
Takeaways & Limitations
Evaluation uses a common ResNet-101 image feature extractor and reports models trained with cross-entropy or CIDEr-D-based sentence-level reward losses.
Abstract
from arXiv · showhide
It is always well believed that modeling relationships between objects would be helpful for representing and eventually describing an image. Nevertheless, there has not been evidence in support of the idea on image description generation. In this paper, we introduce a new design to explore the connections between objects for image captioning under the umbrella of attention-based encoder-decoder framework. Specifically, we present Graph Convolutional Networks plus Long Short-Term Memory (dubbed as GCN-LSTM) architecture that novelly integrates both semantic and spatial object relationships into image encoder. Technically, we build graphs over the detected objects in an image based on their spatial and semantic connections. The representations of each region proposed on objects are then refined by leveraging graph structure through GCN. With the learnt region-level features, our GCN-LSTM capitalizes on LSTM-based captioning framework with attention mechanism for sentence generation. Extensive experiments are conducted on COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GCN-LSTM increases CIDEr-D performance from 120.1% to 128.7% on COCO testing set.
1 Introduction
Image captioning commonly translates visual input into text, but prior CNN-plus-RNN frameworks did not fully study how object relationships support describing images. GCN-LSTM addresses this by modeling semantic and spatial relationships to enrich region representations for sentence generation.
- Image captioning aims to automatically describe image content with complete, natural sentences.
- Prior CNN-plus-RNN captioning frameworks left the use of mutual object correlations and interactions insufficiently studied.
- Visual relationships encode interactions or relative positions between detected objects as subject-predicate-object structures.
- GCN-LSTM builds semantic and spatial graphs over detected regions and uses graph convolutions to enrich their representations.
- Relation-aware region representations are supplied to attention-based LSTM decoders for sentence generation.
- The paper’s contribution is using visual relationships to enrich region-level representations and enhance image captioning.
2 Related Work
Earlier image-captioning work used CNN-RNN sequence models and attention, while visual-relationship research separately studied spatial or semantic relations. GCN-LSTM distinguishes itself by integrating both relation types into attention-based caption generation through graph convolution.
- Modern image captioning predominantly uses CNN-plus-RNN sequence learning to generate flexible sentences.
- Attention mechanisms let captioning models focus on salient objects while generating corresponding words.
- Visual-relationship detection research has examined spatial relations and semantic interactions between objects.
- GCN-LSTM combines object-level visual attention with semantic and spatial relations through GCN-based relation-aware representations.
- The paper identifies exploiting both relation types for image captioning as a novelty not previously explored.
3 Image Captioning by Exploring Visual Relationship
GCN-LSTM represents detected image regions as semantic and spatial graphs, refines them with relation-aware graph convolutions, and decodes captions using attention-based LSTMs.
- Problem Formulation: GCN-LSTM detects salient object regions with Faster R-CNN and constructs separate semantic and spatial graphs over those regions.Vertices represent detected regions, while edges encode semantic or spatial relationships.
- Image Captioning with Visual Relationship: A GCN image encoder contextually refines region features using the constructed semantic or spatial graph before attention-based sentence generation.The captioning framework encodes regions with relation-aware features and feeds them into an attention LSTM decoder.
- Visual Relationship between Objects in Images: Semantic relationships are modeled as directional subject-predicate-object connections between detected object regions.A learned classifier predicts relation classes from the two regions and their union bounding box, then forms the semantic graph.
- Visual Relationship between Objects in Images: Spatial relationships encode relative geometry between object regions using IoU, distance, and angle, with eleven relation types defined for connected pairs.The spatial graph omits pairs whose distance is large and overlap is low.
- Image Captioning with Visual Relationship: The modified GCN uses separate transformation matrices and bias vectors for edge directions and labels, making region encoding sensitive to labeled directional relationships.This extends ordinary neighborhood aggregation to the paper’s semantic and spatial graphs.
4 Experiments
Experiments evaluate GCN-LSTM on COCO using established captioning metrics, qualitative comparisons, human judgments, online testing, and fusion analyses. Across these evaluations, the model outperforms competing approaches, while late graph fusion and a tradeoff value near 0.7 perform best in the reported analyses.
- Datasets and Experimental Settings: Experiments use COCO for image captioning, with Visual Genome supporting pretraining of the object and semantic relation detectors.The COCO setup uses 113,287 training images, 5K validation images, and 5K testing images; Visual Genome supplies densely annotated objects, attributes, and relationships.
- Datasets and Experimental Settings: The evaluation reports BLEU@N, METEOR, ROUGE-L, CIDEr-D, and SPICE scores, with models compared under cross-entropy and expected sentence-level reward optimization.The reward is measured with CIDEr-D, and comparisons use a common ResNet-101 image-feature architecture.
- Performance Comparison and Experimental Analysis: 117.1% CIDEr-D and 21.1% SPICE are achieved with cross-entropy loss, improving over Up-Down by 3.2% and 3.9%, respectively.With CIDEr-D optimization, the reported scores increase to 128.7% CIDEr-D and 22.1% SPICE.
- Performance Comparison and Experimental Analysis: GCN-LSTM achieves the best performances across all evaluation metrics on both c5 and c40 COCO online testing sets.The online submission is optimized with CIDEr-D score and is compared with the latest officially published top-five methods.
- Performance Comparison and Experimental Analysis: Human evaluation reports M2 scores of 92.6% for Human, 82.1% for GCN-LSTM, 78.5% for Up-Down, and 57.8% for LSTM.The study uses 12 evaluators and a randomly selected subset of 1K testing images, assessing resemblance to human captions and human-versus-system discrimination.
- Performance Comparison and Experimental Analysis: 116.4%, 116.6%, and 117.1% CIDEr-D are obtained for early-before-attention, early-after-attention, and late fusion, respectively, making late fusion best.The tradeoff-parameter curves are generally inverted-∧ shaped, with best performance at α about 0.7.
5 Conclusions
GCN-LSTM explores semantic and spatial relationships between detected image regions to enrich representations for caption generation. Experiments on COCO validate the proposal and report new state-of-the-art performance.
- GCN-LSTM models mutual interactions between detected objects or regions to enrich the region-level representations fed to the sentence decoder.
- The architecture captures both semantic and spatial correlations through graph convolutions over detected regions.
- Relation-aware region representations are input to an attention LSTM for sentence generation.
- Extensive COCO image-captioning experiments validate the proposal and analysis, achieving new state-of-the-art performance on the dataset.
- A proposed future direction is generalizing relationship modeling and utilization to other vision tasks.