Source-linked AI summary
Unpaired Image Captioning via Scene Graph Alignments
Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Handong Zhao, Xu Yang, Gang Wang
TL;DR
Image captioning commonly depends on paired image-caption datasets, whose creation is costly and labor-intensive. The paper instead uses scene graphs and unsupervised feature alignment to map image features into the text modality. On MSCOCO, Graph-Align achieves the best performance across all reported metrics among existing unpaired methods, while failure cases reveal information misalignment between image and sentence scene graphs.
Problem
Paired image-caption datasets are costly to create, and unpaired image-to-sentence translation remains difficult because image and text modalities are misaligned.
Method
The framework generates image and sentence scene graphs, trains the scene graph encoder and sentence decoder on text, and learns an unsupervised feature mapping into the text space.
Results
Graph-Align achieves the best performance across all reported metrics against existing unpaired image captioning methods on the MSCOCO test split.
Takeaways & Limitations
Scene graphs provide an effective intermediate representation for connecting images and natural-language descriptions without paired training data.
Takeaways & Limitations
Failure cases arise when image scene graphs focus on local objects while sentence scene graphs convey more information about the images.
Abstract
from arXiv · showhide
Most of current image captioning models heavily rely on paired image-caption datasets. However, getting large scale image-caption paired data is labor-intensive and time-consuming. In this paper, we present a scene graph-based approach for unpaired image captioning. Our framework comprises an image scene graph generator, a sentence scene graph generator, a scene graph encoder, and a sentence decoder. Specifically, we first train the scene graph encoder and the sentence decoder on the text modality. To align the scene graphs between images and sentences, we propose an unsupervised feature alignment method that maps the scene graph features from the image to the sentence modality. Experimental results show that our proposed model can generate quite promising results without using any image-caption training pairs, outperforming existing methods by a wide margin.
1. Introduction
Unpaired image captioning addresses the costly dependence on paired image-caption data, but remains difficult because image and sentence modalities are structurally misaligned. The paper proposes scene graphs and unsupervised feature alignment to bridge these modalities.
- Large-scale paired image-caption data, especially for languages beyond English, requires time-consuming and labor-intensive human expertise.
- Unpaired image-to-sentence translation is less mature than related unsupervised translation settings because image and text encoders cannot simply be shared.The central challenge is information misalignment between images and sentences.
- Prior methods either require pivot-language paired data and parallel translation corpora or use adversarial training with sentence discrimination and visual concepts.Existing approaches relax paired-data requirements but retain additional data requirements or modality-bridging mechanisms.
- Scene graphs abstract objects and their relationships, providing rich semantic information for vision-language tasks.
- The proposed framework combines image and sentence scene graph generators, a scene graph encoder, a sentence decoder, and a feature alignment module.
- The paper contributes a scene graph-based unpaired captioning framework and an unsupervised cross-modal feature alignment method requiring no paired data.Experiments are reported as demonstrating promising captions and superiority over recent unpaired captioning methods.
2. Background
Image captioning has largely used paired image-caption data, while newer unpaired approaches remove direct correspondence but face a harder cross-modal translation problem. This work uses scene graphs explicitly to bridge image and sentence domains.
- Paired Image Captioning: Most established image captioning methods encode images with CNNs and decode captions with recurrent language models under paired supervision.
- Paired Image Captioning: Attention mechanisms improve paired captioning by allowing the decoder to focus dynamically on relevant image regions during generation.
- Paired Image Captioning: Scene graphs have also been used to connect images and sentences in paired captioning, whereas this paper learns the network without paired training data.
- Unpaired Image Captioning: Unpaired image-to-sentence translation is especially challenging because image and text modalities have significantly different characteristics.
- Unpaired Image Captioning: Unlike prior unpaired methods, the proposed approach adopts scene graphs as an explicit representation bridging image and sentence domains.
3. Method
The method represents images and sentences as scene graphs, learns the text-side encoder-decoder first, and then aligns image graph features with the sentence modality for unpaired caption generation.
- 3.2. Unpaired Image Captioning: The framework combines image and sentence scene graph generators, a scene graph encoder, an attention-based sentence decoder, and cycle-consistent feature alignment.An image scene graph is extracted, mapped toward the sentence scene graph, and decoded into a sentence.
- 3.2.4 Training and Inference: The graph encoder and sentence decoder are first trained on text, after which image graph features are mapped into the text feature space for reuse by the decoder.The same sentence decoder can then decode sentences from mapped image features.
- 3.2.1 Scene Graph Generator: Scene graphs contain object, attribute, and relationship nodes, which are encoded using neighborhood-aware spatial graph convolutions.Separate convolutional operations encode object roles, attributes, and relations.
- 3.2.3 Sentence Decoder: Three embedding sets have different lengths and information, so separate attention modules compute relevant context before RNN sentence decoding.The attention vectors are combined into a triplet-level embedding supplied to the decoder.
- 3.2.4 Training and Inference: CycleGAN learns feature alignment between unpaired image- and sentence-scene-graph representations across modalities.The alignment directly accounts for discrepancies between image and sentence graph modalities.
4. Experiments
The experiments evaluate scene-graph encoding, feature alignment, and caption generation under unpaired image-sentence training. Results show that separate attention and adversarial alignment improve performance, while remaining information misalignment causes failures.
- Datasets and Setting: The experiments use Visual Genome to train the image scene graph generator and MSCOCO descriptions to train the sentence scene graph encoder and decoder.The unpaired data for alignment is created by randomly shuffling MSCOCO images and sentences; CIDEr-D, BLEU, METEOR, and SPICE measure caption quality.
- Sentence Decoding: Separate attention mechanisms for object, attribute, and relation features outperform average pooling and shared attention across all reported metrics.The comparison is conducted for sentence decoding on the MSCOCO test split.
- Unpaired Setting without GAN: Graph-Enc-Dec (Att) achieves the best performance among baselines without GAN training, showing that scene graphs can connect images with captions in the unpaired setting.The same results remain inferior to the paired setup, indicating that scene graphs alone do not achieve comparable performance.
- Feature Alignment: The mapping module aligns the three embedding types, particularly attribute embeddings, and aligns triplet features better than raw triplet features.The visualization uses t-SNE scatter diagrams for 1,500 samples.
- Overall Evaluation: Graph-Align achieves the best performance in all metrics against existing unpaired image captioning methods on the MSCOCO test split.Qualitative examples show reasonable descriptions, but failures arise when image scene graphs focus on local objects while sentence scene graphs convey more image information.
5. Conclusions
The paper proposes an unsupervised image captioning framework that uses scene graphs as an intermediate representation and maps image and sentence graph features through cycle-consistent adversarial training. Quantitative and qualitative evaluations show meaningful captions and a margin over existing methods.
- The framework trains image captioning without any paired image-sentence data.
- Scene graphs provide the intermediate representation for images and sentences, with graph convolution and attention encoding objects, attributes, and relationships.
- Cycle-consistent adversarial training maps image and sentence scene graphs in their feature space.
- Quantitative and qualitative evaluations show meaningful captions and performance that outperforms existing methods by a good margin.