Source-linked AI summary
Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs
Shizhe Chen, Qin Jin, Peng Wang, Qi Wu
TL;DR
Most image captioning models do not account for users’ interests in content or desired detail, limiting fine-grained control over descriptions. This paper introduces Abstract Scene Graphs and ASG2Caption, achieving state-of-the-art controllability on two datasets and improving diversity from automatically sampled graphs.
Problem
Most image captioning models are intention-agnostic, generating descriptions without controlling users’ preferred contents or level of detail.
Method
The paper represents user intentions with image-grounded Abstract Scene Graphs containing abstract object, attribute, and relationship nodes, then generates captions with ASG2Caption.
Results
The model achieves state-of-the-art controllability on VisualGenome and MSCOCO, while automatically sampled ASGs significantly improve caption diversity.
Takeaways & Limitations
Abstract Scene Graphs provide fine-grained control over which image contents and detail levels appear in generated captions.
Abstract
from arXiv · showhide
Humans are able to describe image contents with coarse to fine details as they wish. However, most image captioning models are intention-agnostic which can not generate diverse descriptions according to different user intentions initiatively. In this work, we propose the Abstract Scene Graph (ASG) structure to represent user intention in fine-grained level and control what and how detailed the generated description should be. The ASG is a directed graph consisting of three types of \textbf{abstract nodes} (object, attribute, relationship) grounded in the image without any concrete semantic labels. Thus it is easy to obtain either manually or automatically. From the ASG, we propose a novel ASG2Caption model, which is able to recognise user intentions and semantics in the graph, and therefore generate desired captions according to the graph structure. Our model achieves better controllability conditioning on ASGs than carefully designed baselines on both VisualGenome and MSCOCO datasets. It also significantly improves the caption diversity via automatically sampling diverse ASGs as control signals.
1. Introduction
The paper addresses intention-agnostic image captioning by introducing Abstract Scene Graphs (ASGs) for fine-grained control over caption contents and detail. It proposes ASG2Caption, which generates captions from ASGs and improves controllability and diversity on VisualGenome and MSCOCO.
- Problem: Most captioning models passively generate descriptions without modeling users’ preferred contents or level of detail.Humans can describe image contents from coarse to fine details, whereas current systems fail to realize such intentions.
- Proposed control signal: Abstract Scene Graphs represent user intentions as directed graphs of abstract object, attribute, and relationship nodes grounded in the image.The nodes require no concrete semantic labels, making ASGs easy to obtain manually or automatically.
- Model: ASG2Caption uses a role-aware graph encoder and language decoder to recognize abstract graph nodes and generate captions with intended contents and orders.The encoder addresses the need to capture both intentions and semantics when ASGs provide scene layouts without semantic labels.
- Experiments: Experiments on VisualGenome and MSCOCO show better controllability than carefully designed baselines and more diverse captions from automatically sampled ASGs.ASGs are automatically constructed for training and evaluation because no datasets with ASG annotations are available.
2. Related Work
Prior image captioning builds on neural encoder-decoder models and attention, while controllable captioning mainly regulates style or image content. Existing content controls do not provide fine-grained joint control of objects, attributes, relationships, and description order, motivating ASGs.
- Image Captioning: Neural encoder-decoder captioning uses CNNs to encode images and RNNs to generate words, with attention models grounding words in relevant image regions.Show-Tell exemplifies fixed-length image encoding followed by sequential decoding; attentive models dynamically ground generated words with visual parts.
- Controllable Captioning: Controllable text generation conditions sentences on designated signals such as sentiment, style, or semantics to improve interactivity, interpretability, and diversity.For image captioning, prior controls broadly divide into style control and content control.
- Style Control: Style-control methods describe images in different styles, often disentangling style codes from semantic content because paired stylized training texts are scarce.Unpaired style transfer is therefore commonly applied in recent approaches.
- Content Control: Content-control methods target different image aspects, including regions, objects, and object order, but do not jointly control fine-grained attributes, relationships, inclusion, quantity, or description order.Dense captioning describes diverse detected regions, while later methods constrain human-focused objects or multiple objects and their order; ASGs address the remaining gap by controlling objects, attributes, and relationships together.
3. Abstract Scene Graph
The Abstract Scene Graph (ASG) represents fine-grained user intentions as a control signal for customized image caption generation. It uses an unlabeled directed graph whose structure specifies which objects, attributes, and relationships should appear in the description.
- Graph Structure: An ASG for image I is a directed graph G = (V, E), where V and E are its node and edge sets.The graph serves as the control signal for generating customized image captions.
- Graph Structure: ASG nodes have three intention-based types: object nodes o, attribute nodes a, and relationship nodes r.These abstract nodes are grounded in the image without concrete semantic labels.
- User Intention Encoding: Users add grounded object nodes, optional attribute nodes for descriptive detail, and relationship nodes when describing relationships between objects.Multiple attribute nodes may be associated with one object, and relationships are represented through directed edges.
- Automatic Generation: ASGs can be generated automatically from off-the-shelf object proposal networks, optionally using a simple relationship classifier, without externally trained semantic detectors.Diverse automatically extracted ASGs capture different image aspects and support diverse caption generation.
4. The ASG2Caption Model
ASG2Caption encodes abstract scene graphs with role-aware, context-sensitive node embeddings and decodes captions using attention mechanisms that combine graph semantics with intended node order. It also updates node representations to track which graph elements have been expressed during generation.
- 4.1 Encoder: The encoder combines visual features, node-role embeddings, and neighboring-node context to produce ASG node embeddings that reflect both appearance and intention.Object, attribute, and relationship nodes use different grounded visual features and role-specific representations; positional embeddings distinguish attributes connected to the same object.
- 4.1 Encoder: A multi-relational graph convolutional network propagates context through six directed edge types and supports both final node embeddings and a global graph representation.The model adds inverse directions because influence between connected nodes is mutual, then fuses the averaged node embeddings with the global image representation.
- 4.2 Decoder: The decoder combines graph content attention with graph flow attention so caption generation reflects both node semantics and the user-designated order encoded by ASG structure.Graph flow attention can stay on a node or move one or two steps, while a dynamic gate interpolates these flow scores before combining them with content attention.
- 4.2 Decoder: After each generated word, graph updating records node access and uses a visual sentinel to avoid updating nodes accessed only for non-visual words.Node updates decompose into erase and add operations, allowing representations to be zeroed after use while preserving nodes requiring multiple accesses.
5. Experiments
Experiments on VisualGenome and MSCOCO evaluate ASG2Caption for controllability and diversity using aligned or sampled ASGs. The model improves fine-grained attribute control, benefits from its proposed graph components, and generates diverse descriptions that reflect different ASG intentions.
- Experimental Setup: Experiments construct image–ASG–caption triplets from VisualGenome and MSCOCO by parsing captions, grounding objects, and retaining graph layouts and node types without semantic labels.MSCOCO contains more than 120,000 images, each annotated with around five descriptions.
- Evaluation Metrics: Controllability is evaluated with ASG-aligned captions using BLEU, METEOR, ROUGE, CIDEr, and SPICE, while diversity uses Div-n and SelfCIDEr over sampled captions.Higher scores indicate better caption quality or greater diversity, respectively.
- Controllability Results: Controllable baselines outperform intention-agnostic models, while ASG2Caption achieves stronger overall quality and graph-structure alignment across evaluation metrics.Baseline models particularly struggle to generate designated attributes compared with objects and relationships.
- Controllability Results: 0.7 → 0.3 and 1.0 → 0.3 are the reported attribute-control misalignment reductions on VisualGenome and MSCOCO, respectively.These reductions are described as more than half of the attribute-control misalignment.
- Ablation Study: Role-aware node embedding, MR-GCN contextual encoding, graph flow attention, and graph updating each contribute to improved ablation performance.The study specifically finds that distinguishing intention roles and incorporating graph context benefit node encoding, while graph flow attention complements the other decoder enhancements.
- Diverse Caption Generation: Sampling connected ASG subgraphs and attributes produces captions that are more diverse than compared methods, especially on the SelfCIDEr metric.The generated descriptions respect their ASGs, and diverse ASGs lead to significantly diverse image descriptions.
6. Conclusion
The work introduces Abstract Scene Graphs as fine-grained control signals for intention-aware image caption generation, together with an ASG2Caption model for producing desired descriptions.
- Conclusion: Abstract Scene Graphs provide fine-grained control over what image content to describe and how detailed the description should be.Each ASG contains abstract object, attribute, and relationship nodes grounded in the image without semantic labels.
- Conclusion: The proposed ASG2Caption model uses the ASG control signal to support controllable image caption generation that considers user intentions.The passage identifies ASG2Caption as the model proposed for generating desired image descriptions from ASGs.
A. Automatic ASG Generation
Automatic ASG generation detects image regions as abstract object nodes, then adds attributes and meaningful relationships without requiring concrete semantic labels. A trained classifier identifies which object pairs should be connected, enabling construction of a global ASG during inference.
- Automatic ASG Generation: Off-the-shelf object proposal models detect candidate regions as abstract object nodes, while attribute and relationship nodes can be added without semantic labels.Attributes attach to objects, and relationships connect object pairs; arbitrary relationships are not necessarily meaningful or common.
- Automatic ASG Generation: The relationship classifier predicts three classes: no relationship, subject-to-object relationship, or object-to-subject relationship.It uses global image appearance and the visual features of each object region for prediction.
- Automatic ASG Generation: During inference, SoftNMS reduces redundant detected boxes before the classifier evaluates every object pair and builds a global ASG.Pairs are considered meaningfully related when the no-relationship probability is below 0.5, with direction selected from the predicted relationship classes.
B. ASG Dataset Construction
The ASG dataset is constructed differently for VisualGenome and MSCOCO because only VisualGenome provides grounded region scene graphs, though they are noisy. For MSCOCO, ASGs are built from parsed descriptions, detected objects, and grounding procedures, despite some construction errors.
- Dataset-specific construction: VisualGenome ASGs use existing region scene graphs as references because those graphs contain missing objects, relationships, and misaligned attributes.MSCOCO has no grounded scene graphs, so its ASGs must be built from scratch.
- Construction pipeline: ASG construction first parses each description with the Stanford scene graph parser to obtain semantic labels, node types, and connections.The parsed scene graph provides the structure later grounded in image regions.
- Construction pipeline: Candidate object regions come from annotated VisualGenome boxes or a Faster-RCNN detector pretrained on VisualGenome for MSCOCO.The detector supplies candidate object bounding boxes and labels for MSCOCO images.
- Construction pipeline: Objects in the parsed scene graph are grounded to candidate bounding boxes to construct the image-description ASG pairs.The grounding step links parsed graph objects with image regions.
- Construction errors and robustness: Automatic construction can misparse non-objects as objects or ground one object label to only part of its depicted instances, but most constructed pairs are correct.These imperfections nevertheless allow the model to learn from the dataset.
C. Graph Structure Metric
The Graph Structure metric measures alignment of tuple structures between generated and ground-truth captions to evaluate ASG structure control, excluding semantic correctness.
- C. Graph Structure Metric: The metric adapts SPICE’s tuple-based parsing to assess structural alignment without evaluating semantic correctness.SPICE parses object, object-attribute, and object-relationship-object tuples, whereas Graph Structure focuses only on their structure.
D. Additional Qualitative Results
The paper presents additional qualitative examples of controllable caption generation with designated ASGs and diverse caption generation with sampled ASGs. It also identifies three main ASG2Caption error types and notes that multiple requested attributes can cause repetition or incorrect attribute generation.
- Error Analysis: The model’s three main mistake types are object recognition, relationship detection, and attribute generation errors.Attribute generation errors mostly occur when multiple attributes are required, producing repeated or incorrect attributes.
- Qualitative Examples: Figure 9 presents controllable image caption examples conditioned on designated ASGs, alongside ground-truth and C-BUTD captions.The examples compare the proposed model with the groundtruth and the state-of-the-art model C-BUTD [3].
- Qualitative Examples: Figure 10 provides examples of diverse image caption generation conditioned on sampled ASGs.These examples illustrate diverse caption generation using automatically sampled control graphs.