Source-linked AI summary
Comprehensive Image Captioning via Scene Graph Decomposition
Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, Yin Li
TL;DR
Image captioning methods often prioritize caption quality, with generic sentences and irrelevant-region attention remaining concerns, while diversity, grounding, and controllability have been addressed separately. This paper decomposes image scene graphs into sub-graphs and selects and decodes them into captions. It reports state-of-the-art results in diversity, grounding, and controllability, and favorable comparison with recent methods on caption quality.
Problem
Many captioning approaches prioritize quality and can produce generic sentences or attend to irrelevant regions, while no prior method addresses diversity, grounding, and controllability together.
Source: §1 · pp. 1–2Method
The model decomposes an image scene graph into candidate sub-graphs, selects meaningful ones with a graph neural network, and decodes each selected sub-graph into a sentence.
Source: §1 · p. 2Results
The model establishes state-of-the-art results in caption diversity, grounding, and controllability, and compares favorably with recent methods on caption quality.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
Selecting distinct scene-graph sub-graphs supports diverse caption generation, while specifying sub-graphs supports controllable sentence generation.
Source: §1 · p. 2The model assumes a sub-graph can be used to decode its target sentence; otherwise, the sentence probability is zero.
Source: §3 · p. 4
Abstract
from arXiv · showhide
We address the challenging problem of image captioning by revisiting the representation of image scene graph. At the core of our method lies the decomposition of a scene graph into a set of sub-graphs, with each sub-graph capturing a semantic component of the input image. We design a deep model to select important sub-graphs, and to decode each selected sub-graph into a single target sentence. By using sub-graphs, our model is able to attend to different components of the image. Our method thus accounts for accurate, diverse, grounded and controllable captioning at the same time. We present extensive experiments to demonstrate the benefits of our comprehensive captioning model. Our method establishes new state-of-the-art results in caption diversity, grounding, and controllability, and compares favourably to latest methods in caption quality. Our project website can be found at http://pages.cs.wisc.edu/~yiwuzhong/Sub-GC.html.
1 Introduction
The paper targets captioning that can describe distinct scene components while linking sentence tokens to image regions. It represents images with scene graphs and selects sub-graphs to generate component-specific captions, aiming to combine caption quality, diversity, grounding, and controllability.
1 Introduction
- A single image can convey multiple ideas through different scene components, and people can link the words in a description back to image regions.The examples include a boy flying a kite, a ship sailing on a river, and a bike parked on the street.Source: §1 · p. 1
- Existing approaches prioritize caption quality and can produce generic sentences or attend to irrelevant regions, while prior efforts address diversity, grounding, or controllability separately.The paper identifies the lack of a method addressing all three capabilities together.Source: §1 · pp. 1–2
- Scene graphs are decomposed into candidate sub-graphs, which a graph neural network selects and decodes into sentences using node attention for token-to-region backtracking.Objects are nodes and relations between object pairs are edges; selecting multiple sub-graphs supports diverse captions, while specifying sub-graphs supports controllable generation.Source: §1 · p. 2
- The model reports strong grounding and improved state-of-the-art controllability, while its top-ranked captions compare favorably with recent caption-quality methods and outperform diverse-captioning methods on quality and diversity metrics.The reported evaluations use MS-COCO Caption and Flickr30K Entities.Source: §1 · pp. 2–3
2 Related Work
Related work spans conventional encoder-decoder captioning, scene-graph methods, grounded and localized descriptions, and approaches to diversity and controllability. This paper distinguishes its approach by selecting scene-graph sub-graphs and generating sentences from them using image-sentence pairs.
2 Related Work
- Conventional captioners commonly combine CNN image features with RNN sentence decoding, while object-detector features have also been used to improve caption quality.Object information has been shown important for captioning.Source: §2 · p. 3
- Prior scene-graph captioning aggregates information over the full graph, whereas this method selects ranked sub-graphs to produce diverse and controllable sentences.The paper uses a graph convolutional network for the input scene graph, as does.Source: §2 · p. 3
- Grounded captioning aligns generated words with image regions, while dense captioning describes local regions or region pairs; this method targets scene components spanning multiple regions.Its grounded-captioning setting assumes only image-sentence pairs, and sub-graphs constrain grounding to the selected graph component.Source: §2 · p. 3
- Diverse-captioning work includes latent-variable and part-of-speech-controlled approaches, while controllable methods use templates or grounded regions.Unlike, this method trains from image-sentence pairs and also provides capacity for caption grounding; it applies scene-graph decomposition to captioning, a setting not previously explored for graph partitioning.Source: §2 · p. 4
3 Method
The method generates captions for different image components by decomposing an image scene graph into sub-graphs, selecting meaningful sub-graphs, and decoding them into grounded sentences.
3.1 Scene Graph Detection and Decomposition
- MotifNet represents objects as nodes and relations as directed edges; neighborhood sampling forms sub-graphs intended to cover potential scene components.Source: §3.1 · p. 5
3.2 Sub-graph Proposal Network
- The sGPN combines node visual and text features with predicate embeddings, uses GCN context, and scores sampled sub-graphs for captioning.Source: §3.2 · pp. 5–6
- Human captions supervise binary good/bad sub-graph labels via noun-matched reference-graph node IoU; positives require IoU above 0.75.Training batches balance positive and negative sub-graphs at 1:1, and use a reference sub-graph as positive when none sampled matches.Source: §3.2 · p. 7
3.3 Decoding Sentences from Sub-graphs
- An attention LSTM weights selected sub-graph nodes at each step; a language LSTM decodes words, while attention links tokens to image regions.Source: §3.3 · pp. 7–8
3.4 Training and Inference
- Training uses binary cross-entropy for sGPN and multi-way cross-entropy for sentence decoding, with the losses weighted equally.Source: §3.4 · p. 8
- Greedy NMS filters highly overlapping sub-graphs; top-ranked survivors are decoded, allowing sub-graph choice to control caption content.The authors report that NMS helps remove redundant captions and promote diversity; attention weights ground decoded tokens to image regions.Source: §3.4 · p. 8
4 Experiments
The experiments evaluate Sub-GC on accurate and diverse, grounded, and controllable captioning. Results show strong performance across these tasks, with comparisons favorable to recent accuracy-focused methods.
4 Experiments
- The experiments use Faster R-CNN and Motif-Net scene graphs, sample sub-graphs, and decode sentences with a two-layer GCN and attention LSTM.For each image, the setup keeps 36 objects and 64 triplets, samples 1000 sub-graphs, and removes duplicates; beam size is 2 unless otherwise noted.Source: §4 · pp. 8–9
4.1 Ablation Study
- sGPN outperforms consensus re-ranking in accuracy and diversity, while lower NMS thresholds and top-K sampling increase diversity with accuracy trade-offs.With NMS=0.75, sGPN gains +1.1 CIDEr and +12% 1-gram over consensus; with T=0.6, K=3, top-K sampling changes CIDEr by -3.6 and 1-gram diversity by +22%.Source: §4.1 · pp. 9–10, Table 1 · p. 9
4.2 Accurate and Diverse Image Captioning
- +2.6/2.1 in B4 and +11.2/9.9 in CIDEr at 20/100 samples over previous best results for Sub-GC.Sub-GC also achieves +8% novel sentences and +29%/20% for 1/2-gram diversity with 20 samples.Source: §4.2 · pp. 10–11, Table 2 · p. 11
- Sub-GC is within 1–2 points in B4/CIDEr of recent accuracy-focused methods, while Sub-GC-oracle exceeds all compared methods by +22.4 in B4 and +50.0 in CIDEr.The authors infer that at least one high-quality caption exists among sentences decoded from sub-graphs, if a way to select that sub-graph is available.Source: Table 3 · p. 11, §4.2 · p. 12
4.3 Grounded Image Captioning
- +1.0 F1all and +3.0 F1loc over previous best weakly supervised results, with caption quality also higher by +1.1 in B4 and +0.5 in CIDEr.Against Full-GC, Sub-GC is lower by -0.6 in B4 and -1.6 in CIDEr, but higher by +1.1 in F1all and +3.5 in F1loc.Source: Table 4 · p. 12, §4.3 · p. 13
4.4 Controllable Image Captioning
- Supervised Sub-GC exceeds previous supervised methods by +1.3 in B4, +6.4 in CIDEr, and +1.4 in IoU; image-sentence-only Sub-GC performs comparably to prior methods.Both variants decode from best-matching sub-graphs for input region sets; the supervised variant also trains on region-set/sentence matches.Source: Table 5 · p. 12, §4.4 · p. 14
5 Conclusion
The paper concludes that selecting important scene-graph sub-graphs and decoding one sentence from each supports accurate, diverse, grounded, and controllable image captioning. It reports state-of-the-art results in diversity, grounding, and controllability, and favorable comparison on caption quality.
5 Conclusion
- Sub-GC selects important scene-graph sub-graphs and decodes one target sentence from each, supporting accurate, diverse, grounded, and controllable captions.The authors describe it as the first comprehensive model for image captioning.Source: §5 · p. 14
Appendix A Additional Implementation Details
The appendix specifies additional network and inference implementation details.
Appendix A Additional Implementation Details
- The appendix sets f(·) to map 2048D inputs through 512D to a scalar, and g(·) to map 2048D inputs through 512D back to 2048D.GCN layers transform features to 1024D; consensus re-ranking uses global image features from ResNet-101 pretrained on ImageNet.Source: Appendix A · p. 20
Appendix B More Qualitative Results
The appendix illustrates multiple captions generated from different sub-graphs and shows how decoded tokens are grounded in image regions.
Appendix B More Qualitative Results
- Figures 4–7 show multiple top-ranked sub-graphs decoded into sentences and associated with image-region groundings.Each row represents one sub-graph; visualized graph nodes have maximum attention weights for decoded tokens.Source: Appendix B · p. 20
- For one image, a man-and-street sub-graph yields a walking caption, while a bus-and-building sub-graph yields a separate parked-bus caption, with nouns linked to image regions.The examples show that different selected scene components can support distinct descriptions of the same image.Source: Appendix B · p. 20, Figure 4 · p. 21
- Figures 4–7 arrange the input image and detected objects, sub-graph, and grounded sentence from left to right, with decoded nouns and regions color-matched.The visualizations also show edges between the nodes with maximum attention weights.Source: Appendix B · p. 20, Figure 4 · p. 21, Figure 5 · p. 22